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Foreword 



This report is the result of work over the last year by the CAISE Publie Partieipation in Seientifie 
Researeh Inquiry Group. We are grateful to all of the members of the group for their eontributions and to 
Riek Bonney of the Cornell Lab of Ornithology for serving as lead author of this report. 

CAISE Inquiry Groups help to strengthen and eonneet the informal seienee edueation eommunity by 
eatalyzing eonversations aeross the field around issues and topies of eornmon eoneern. Drawing in part 
on results from NSF-funded projeets, programs, and produets, Inquiry Groups meet together to examine 
and diseuss an issue or set of questions, gather and analyze evidenee from praetiee and researeh, and 
synthesize their findings. They doeument their work with three audienees in mind: the informal seienee 
edueation (ISE) field as a whole, prineipal investigators (Pis) and prospeetive Pis of projeets funded 
by NSF’s Informal Seienee Edueation (ISE) program, and ISE program offieers. Inquiry Groups have 
multiple uses. Their work produets are intended to yield praetieal knowledge that ean be put to work, 
provide evidenee of eontributions of ISE, and identify areas where more work and investment are 
needed. 

The diseussions that began among the members of the Publie Partieipation in Seientifie Researeh Inquiry 
Group are intended to inform and spark further study, diseussion, and refieetion among eolleagues from 
aeross the field. We look forward to eontinuing the eonversation. To find out about online diseussions 
and eonferenee sessions, visit the CAISE web site (www.eaise.insei.org) and subseribe to the CAISE 
Newsletter. 

We are grateful to the National Seienee Foundation for its support of CAISE and the informal seienee 
edueation field. 

Wendy Polloek 
Prineipal Investigator 

Center for Advaneement of Informal Seienee Edueation 
Washington, D.C. 
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management. She received her Ph.D. in Environmental Science, Policy, and Management from the 
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RcbCCCa Jordan is an Assistant Professor of Environmental Education and Citizen Science in the 
Department of Ecology, Evolution, and Natural Resources at Rutgers University New Brunswick. As 
director of the Program in Science Learning, Rebecca is a behavioral biologist who devotes most of her 
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curricula. She received her M.Sc. and Ph.D. in Organismic and Evolutionary Biology at the University 
of Massachusetts and completed a post-doctoral fellowship at Princeton University and a visiting 
assistant professorship at Elizabeth City State University in partnership with the University of North 
Carolina at Chapel Hill. 

Ellen McCallie is Deputy Director of the Carnegie Museum of Natural History in Pittsburgh, and is 
completing a Ph.D. as part of the Center for Informal Learning and Schools (CILS) at King’s College 
London. Ellen’s research seeks to facilitate interaction, collaboration, and meaning-making among 
scientists, publics, evaluators, and ISE professionals. Previously, Ellen served as Director of the Center 
for Advancement of Informal Science Education (CAISE). Ellen has also worked in various ISE media, 
including museums, gardens, and science television. She began her career conducting ecological and 
agricultural research in Brazil and Indonesia. 
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Tina Phillips is an Extension Associate at the Cornell Lab of Ornithology and Project Leader for 
NestWatch, NestCams, and CamClickr. She received a B.S. in Biology from the State University of New 
York at Stony Brook where she conducted research studying animal behavior. Lor the past decade, she 
has worked in the Lab of Ornithology’s Citizen Science Program using citizen-collected data to study 
bird breeding biology. She has helped to develop numerous science inquiry curricula, and frequently 
writes and speaks about the educational and scientific outcomes of citizen science. In addition, she is 
pursuing a Ph.D. in science education at Cornell University, examining the relationship between citizen 
science participation and environmental literacy and behavior. 

Jennifer Shirk works at the Cornell Lab of Ornithology as the Citizen Science Toolkit Project Leader. 
She coordinated the 2007 Citizen Science Toolkit Conference, and currently facilitates a website to 
support and network Public Participation in Research projects (www.citizenscience.org), which was 
inspired by the needs of conference participants for shared resources and strategies across different 
programmatic models. Shirk received her M.S. in 2005 from the Cornell Department of Natural 
Resources for investigating an inquiry-based School-Science Partnership, and is currently a Ph.D. 
candidate in the same department working to understand the strategies employed by science researchers 
to find professional success through Public Participation in Research. 

Candic C. Wildcrman is a Professor of Environmental Sciences at Dickinson College in Carlisle, 
Pennsylvania and Chair of the Environmental Studies Department. She received an M.A. in geology 
from Harvard University and a Ph.D. in Geography and Environmental Engineering from Johns 
Hopkins University. She is Lounder and Science Director of ALLARM (Alliance for Aquatic Resource 
Monitoring), a community-based volunteer stream monitoring network in Pennsylvania, which was 
founded in 1986 and which provides technical and programmatic mentoring support to watershed 
groups and communities involved in watershed documentation, protection, and restoration. Her 
teaching and research interests include operational models for community-based research and education, 
undergraduate pedagogy, watershed assessment and management, aquatic ecology, and Chesapeake Bay 
restoration and protection issues. 
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Executive Summary 



Introduction 

This report describes how Public Participation in Scientific Research (PPSR), in the context of 
informal science education (ISE), can provide multiple opportunities to increase public science 
literacy. 

The ISE field is large and continually evolving. In the broadest sense it encompasses the range of 
science learning opportunities and activities that people experience across their lifespan outside 
of school. ISE can be delivered via numerous venues including lectures, TV programs, films, 
exhibits, websites, digital games, and community projects that are experienced or viewed in homes, 
science centers and museums, zoos and aquariums, botanical gardens and nature centers, and youth, 
community, and after-school centers. 

As the concept of ISE emerged in the 1940s and 50s it was called “Public Understanding of Science” 
(PUS), and the current ISE program at NSF, which is one of the largest funders of ISE project 
development in the United States, is a direct descendant of the Public Understanding of Science 
program created at NSF in 1958 (Eewenstein 1992). Many ISE projects still follow the PUS concept 
or model, which is premised on the idea that science, scientists, and other experts know and should 
determine what the public needs to learn. Explain science to the public, the reasoning goes, and 
both science and scientists will enjoy greater support, which in turn will lead to greater economic 
prosperity, enhanced quality of life, and world leadership in science and technology. Most PUS 
activities involve exhibits, lectures, media broadcasts, and public programs through which the public is 
informed about science and expected to embrace it. 

Recent research has suggested several shortcomings in the PUS model, two of which are relevant to 
this inquiry. First, educational research shows that people have greater motivation to engage and learn 
if the subject matter is directly relevant to their lives and interests and/or if the learning process is 
interactive — one in which the learner can directly affect the learning process, content, and/or outcomes 
of the experience (Falk 2001). 

Second, PUS usually focuses on delivery of specific content rather than on helping the public 
experience and understand the process of research, that is, the way that scientific questions are asked, 
answered, and debated by the scientific community (Eewenstein and Bonney 2004). 

To address these concerns, many ISE programs, projects, and activities developed over the past two 
decades have aspired to actively involve the public directly in the multifaceted and iterative processes 
of scientific investigation. Such efforts include citizen science, volunteer monitoring, and participatory 
action research. Projects that fall into these categories allow participants to learn both science content 
and process while experiencing the fun and excitement of research. 

In response to a request by the National Science Foundation, the Center for Advancement of Informal 
Science Education (CAISE) established an Inquiry Group to 

• identify and describe the range of ISE projects and activities in which the public is involved in 
one or more of the various stages of research 

• describe models for public participation in scientific research 
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• understand and describe the educational impacts of PPSR projects 

• make recommendations for conceptualizing and developing future ISE activities that will 
enhance public participation in scientific research. 

We hope that this report will serve as a starting point for discussion about the value and potential for 
PPSR projects as a form of informal science education. While we have attempted to capture current 
thinking in this area, we realize that the conversation is just beginning. 

Models for Public Participation in Scientific Research 

Scientific investigations include many processes, steps, or activities in which the public can be 
involved. These include: 

• Choosing or defining questions for study 

• Gathering information and resources 

• Developing explanations (hypotheses) about possible answers to questions 

• Designing data collection methodologies (both experimental and observational) 

• Collecting data 

• Analyzing data 

• Interpreting data and drawing conclusions 

• Disseminating conclusions 

• Discussing results and asking new questions 

From an educational perspective, PPSR models differ chiefiy by involving the public in these steps to 
varying degrees and by altering the amount of control that participants have over the different steps. 
For this report we have divided PPSR projects into three major categories: 

1) Contributory projects, which are generally designed by scientists and for which members of 
the public primarily contribute data 

2) Collaborative projects, which are generally designed by scientists and for which members 
of the public contribute data but also may help to refine project design, analyze data, or 
disseminate findings 

3) Co-created projects, which are designed by scientists and members of the public working 
together and for which at least some of the public participants are actively involved in most or 
all steps of the scientific process. 

Impacts of Public Participation in Scientific Research 

To investigate the educational impacts of PPSR we reviewed ten projects including examples of each 
model type — five Contributory, three Collaborative, and two Co-created. To standardize our methods 
of analysis for each selected project, we developed a rubric based on the evaluation framework 
described in Evaluating Impacts of Informal Science Education Projects (Friedman 2008). Our rubric 
describes potential impacts in the general categories of developing understanding and knowledge, 
enhancing engagement or interest, improving skills, changing attitudes, and changing behavior. 
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Next, for each project that we reviewed, we examined project descriptions and published reports to 
identify goals, objectives, and potential indicators for each impact category. Finally we determined 
measured outcomes (which were rare for most studies) and inferred outcomes (outcomes that seem to 
be happening but which are based on anecdote or perception as opposed to qualitative or quantitative 
evaluation). 

In most cases, using this rubric to assess project impact was challenging because few if any of the 
projects that we reviewed used anything like it in their original or ongoing evaluations. However, we 
believe that our rubric represents an important step in bringing cohesion to the emerging PPSR field. 

It can be used not only as a guide to project assessment, but also as a tool for project planning and 
development. 

Awareness, knowledge, and/or understanding 

All PPSR projects seem to contribute to awareness, knowledge, and/or understanding of key 
scientific concepts related to the study at hand. This conceptual understanding ranges from purely 
scientific information to environmental issues and regulations. Participants in many PPSR projects 
also gain knowledge of the process of science. Indeed, this is one area where PPSR projects have the 
potential to yield major impacts, particularly Collaborative and Co-created projects, which engage 
participants in project design and data interpretation to a significant degree. Participants in most of the 
Collaborative and Co-created projects also seem to have gained knowledge of community structure 
and environmental regulation. 

Engagement or interest 

Increasing public engagement in scientific activities is an area at which PPSR projects excel. 
Participants who take part in a project’s full range of activities are deeply engaged in conducting 
science. Indeed, enlisting people into PPSR projects is probably one of the most expedient methods for 
informal science educators to engage people in science in a fun and meaningful way. PPSR projects 
provide opportunities for people to develop interest and engagement by either trying something new or 
by expanding previously existing interests. The different PPSR models allow varied levels and types 
of engagement so that individuals with varying levels of expertise can develop a new interest or take 
deeper steps in their practice or engagement. 

Skills 

PPSR projects are excellent for developing science-related skills. Participants in most projects 
increase their ability to identify organisms, to use measurement instruments, to collect field data 
following specific protocols, and to sample consistently over time. Thousands of individuals have 
submitted data on nesting birds, the distribution and abundance of monarch butterflies, patterns of 
water quality, and the distribution and abundance of invasive plants. Furthermore, these data have 
been of sufficient quality to allow scientific analyses and to be published in peer-reviewed scientific 
publications. Participants in data analysis workshops gain additional skills in reading and interpreting 
graphs, drawing conclusions from evidence, and raising new questions as a basis for new study 
designs. Participants in Collaborative and Co-created projects also increase their ability to identify and 
distinguish habitats, to identify and select study sites, to weigh the pros and cons of various research 
design and data collection methods, and to communicate their results to the public. 

Attitudes and behaviors 

Few instances where PPSR project participation has affected attitudes toward science have been 
documented. Nevertheless, individuals who learn to function as scientists, or at least to understand 
how scientists work, could be expected to increase their already positive attitudes toward science. 

We do see evidence that Co-created projects that are initiated to meet specific community needs can 
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draw concerned citizens into the scientific process who might not otherwise be involved in science- 
related activities. In addition, participation in PPSR projects can change behaviors. First of all, the 
projects cause people to take part in scientific research, which is a huge behavior change for many! 
Participation also can lead to other types of behavior change such as improving habitat for wildlife or 
notieing invasive species in the environment. And some PPSR partieipants become more engaged in 
community politics and more confident about asking for a place at the table in making decisions about 
community planning. 

Opportunities for the Field 

We suggest that projects focusing on PPSR represent an emerging field of informal science education 
that is ripe for further development, and we urge developers of ISE programs to consider four PPSR 
approaches in their work. 

Creating new PPSR projects 

Although hundreds of PPSR projects are now under way, both large and small, room for new projects 
exists in three areas; 

1) Projects designed to study new scientific questions 

2) Projects designed to engage new audiences 

3) Projects designed to test new or enhanced PPSR models. 

Many new projects could be created that will appeal to the increasing numbers of amateur naturalists 
and stargazers who are interested in lending their brains to science. Developers should be aware, 
though, that coming up with questions that are appropriate for PPSR is not a simple task, especially if 
the project desires to reach a wide audience. 

And from an educational perspective, developing projects to answer new scientific questions may 
not be the most strategic approach for the ISE field unless new projects strive to enhance existing 
PPSR strategies or reach truly new audiences. But to reach new audiences, significant research into 
motivations for members of the public to understand and participate in research will be needed. Also 
required is research into the ways in which individuals perceive themselves or can develop identities 
as scientists, potential scientists, and eritical thinkers. We suggest that new projects, especially if they 
desire to engage diverse audiences, test Collaborative and Co-created approaches to PPSR. 

Enhancing PPSR projects already underway 

We believe there is tremendous potential to enhance projects that are already underway. In particular, 
we suggest that many projects could be expanded to involve more individuals in more aspects of 
science inquiry. Eor instance. Contributory projects can be expanded into Collaborative models, 
at least for some project participants. At the same time. Collaborative and Co-created projects can 
learn many lessons from successful Contributory projects. For example, water quality monitoring 
projects could work together to design and develop regional and national databases that would allow 
participants to compare data and findings across larger regions or over long spans of time. 

Add PPSR to other types of ISE projects 

Examination of the NSE portfolio shows that relatively few full-fiedged PPSR projects have been 
funded to date, although several have started in the last few years. More common in the portfolio are 
PPSR “add-ons,” that is, citizen science projects that are associated with other projects such as films 
or exhibitions. We suggest that more closely integrating PPSR efforts into other types of ISE projects 
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would represent strategie investment by the field. ISE projeets seeking to deepen their impact through 
inclusion of PPSR techniques should consider partnering with PPSR projects already under way. 

Enhancing research and evaluation surrounding models of PPSR and their impacts 

Many questions remain to be answered about the potential for PPSR projects to contribute significantly 
to the ISE field. These include questions about personal and extrinsic motivators, altruistic intentions, 
and the notion that projects may be tiered or might serve as gateways for other projects to embrace 
participants who join projects for different reasons. Questions also remain about the learning 
implications of various types of PPSR projects. What is the nature of questions that enable participants 
to not only collect data but also to refiect on their experiences? Do participants, or potential 
participants, prefer to become involved with questions that deal with basic science, that resolve issues, 
that are simply fun, or all of the above? To what extent do participants gain from projects because they 
help to shape them? Yet other questions relate to the overall impacts of PPSR participation, including 
participation in areas of inquiry that have not been well studied such as modeling, data visualization, 
and data dissemination. Einally, we need to better understand the behavioral impacts of PPSR, which 
could push the boundaries of what we currently define as learning in the realm of science, including 
learning that affects participants’ lives in a very broad sense. 
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Part 1 : Introduction 



Members of the public participate in scientific research in many ways and for many reasons. Some 
people conduct personal research to answer their own questions. However, most individuals who 
are involved in scientific research work as part of an organized group or project. Research groups 
involving the public can be as small as a few people gathering data on the quality of a local stream or 
as large as several thousand people participating in a national bird count. 

At the core of most projects is a scientific question or environmental issue that is best addressed by 
analyzing large amounts of data that are collected across a wide area, or over a long period of time, or 
both. For example, understanding how water quality changes across a watershed, or how the timing 
of plant reproduction is affected by a warming climate, requires many individuals to work in concert, 
collecting and contributing information sufficient to understand the issue. 

Public participation in scientific research is not a new concept. Lighthouse keepers began collecting 
data about bird strikes as long ago as 1880. A group of amateurs started the Astronomical Society 
of the Pacific in 1889; the National Weather Service Cooperative Observer Program began in 1890; 
and the National Audubon Society started its annual Christmas Bird Count in 1900 (Droege 2007). 
Throughout the 1900s, individuals across North America participated in projects to monitor water 
quality, document the distribution of breeding birds, and scour the night skies for new stars and even 
galaxies. 

The interest of Science, Technology, Engineering, and Math (STEM) educators in public research 
projects is fairly recent, however, and corresponds roughly with publication of the Benchmarks for 
Scientific Eiteracy (AAAS 1993) and the National Science Education Standards (National Research 
Council 1996). These publications espoused the idea that the public, in addition to knowing science 
facts, should also understand the process by which scientific investigations are conducted. As a result, 
the 1990s saw the emergence of several projects seeking to enlist the public in authentic scientific 
research. 

At the National Science Eoundation, the concept of developing public understanding of science 
through public participation in scientific research seems to have begun with the 1992 award for a 
project called “Public Participation in Ornithology” (DRL-9 155700). Developed by the Cornell Lab 
of Ornithology (CLO), the project launched three “National Science Experiments,” parts of which 
continue today (Bonney 2007). In each of these experiments the public gathered and submitted 
data to answer a specific question following a protocol developed by CLO staff. In addition, each 
project supplied participants with “Research Kits” containing educational support material such as 
identification guides, recorded bird sounds, and background reading explaining the reasons for the 
projects and describing the scientific method. One newspaper reporter, writing about CLO’s Seed 
Preference Test, called the project a “Trojan Horse approach to education.” 

In 1995 Rick Bonney coined the term “Citizen Science” to refer to CLO’s growing number of 
scientist-driven public research projects (Bonney 1996). (At the time, he was unaware of the use of 
the term by Irwin [1995] to refer to citizen engagement in science policy.) The name took hold, and by 
2001 had made its way into NSE’s Informal Science Education' program solicitation. Today, hundreds 
of citizen science projects, a number of which have been sponsored by NSE, are under way across the 
country. Project descriptions for many of them can be found at the website www.citizenscience.org. 



■ A note on terminology: To differentiate when we are referring to the National Science Foundation Informal 
Science Education Program as opposed to the field of infonual science education, we have adopted the following 
convention: The “NSF ISE Program” is used to refer to the former, while “ISE” and “the ISE field” are used for the latter. 
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Citizen science is not the only model for public involvement in research, however. Other models 
include volunteer monitoring, community science, and participatory action research (Cornwall and 
Jewkes 1995, Wilderman et al. 2004, Lawrence 2006, Cooper et al. 2007, Ely 2008). Often these 
models provide participants with a more comprehensive exposure to scientific methodology than 
do the projects typically operated by science institutions. For example, in most participatory action 
research, participants help to ask the research question, design the study, and interpret results in 
addition to collecting data. The various models and terms often blur, however, and defining each one 
precisely is challenging. 

Nevertheless, elucidating the basic models for public participation in scientific research (PPSR) and 
understanding how the models differ is an important first step in describing the emerging PPSR field 
and assessing and anticipating the learning impacts of the entire spectrum of PPSR projects. 
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Part 2: Models for Public Participation in Scientific Research 

Scientific investigations include many processes, steps, or activities in which the public can be 
involved. From an educational perspective, PPSR models differ chiefiy by involving the public in 
various steps to varying degrees and by altering the amount of control that participants have over the 
different steps.^ For this report we have divided PPSR projects into three major categories: 

1) Contributory projects, which are generally designed by seientists and for which members of the 
public primarily contribute data 

2) Collaborative projects, which are generally designed by seientists and for which members 
of the public contribute data but also may help to refine project design, analyze data, or 
disseminate findings 

3) Co-created projects, which are designed by scientists and members of the public working 
together and for whieh at least some of the public participants are actively involved in most or 
all steps of the scientific process. 

The basic models of PPSR are shown in Table 1. 



Table 1 . Models for Public Participation in Scientific Research 



Step in Scientific Process 


Steps included 
in Contributory 
Projects 


Steps included 
in Collaborative 
Projects 


Steps included 
in Co-created 
Projects 


Choose or define 
question(s) for study 






X 


Gather information and 
resources 






X 


Develop explanations 
(hypotheses) 






X 


Design data collection 
methodologies 




(X) 


X 


Collect samples and/or 
record data 


X 


X 


X 


Analyze samples 




X 


X 


Analyze data 


(X) 


X 


X 


Interpret data and draw 
conclusions 




(X) 


X 


Disseminate conclusions/ 
translate results into action 


(X) 


(X) 


X 


Discuss results and ask 
new questions 






X 



X = public included in step; (X) = public sometimes included in step 



^ The authors are not implying that all scientific research includes all of these steps or that there is a defined “order” 

to scientific investigation. Rather, we articulate these steps to outline a range of common research activities in which mem- 
bers of the public might participate. 

• • • 
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Contributory PPSR Project Model 

Contributory projects are researcher-driven data-collection projects. Scientists 
ask questions for which answers require the accumulation of large amounts of data 
collected over wide geographic areas and/or over long spans of time, and members of 
the public collect relevant data following protocols predetermined by the scientists. 
Most projects labeled “citizen science” fall into this model (Bonney 1996, Krasny and 
Bonney 2005, Bonney 2007). In some projects the public is encouraged to analyze data, 
typically using online data visualization tools, but seldom are project participants 
expected to present results of analyses in such a way that they contribute to the 
research process. Participants do not determine any of the steps in Contributory 
projects. An example of a Contributory project is CLO’s Project FeederWatch, for which 
participants count the kinds and numbers of birds that visit backyard feeders during 
winter and report their data on carefully designed data forms. Researchers at CLO then 
analyze the data to understand and report trends in winter bird populations. 

Collaborative PPSR Project Model 

In Collaborative projects, as in Contributory projects, scientists ask research questions, 
and members of the public collect data to help answer those questions. However, 
project participants are actively involved in multiple research activities, including 
analyzing samples, interpreting data and drawing conclusions, and presenting 
results to other members of the public and/or to scientists and community decision 
makers. Participants also may help to design and refine data collection protocols. 

Many volunteer monitoring projects fall into this category (Whitelaw et al. 2003). 
Examples include some of the more complex water quality monitoring projects, for 
which participants work under the guidance of scientists to gather water samples, 
examine their content, perform basic data analyses, and present their findings to local 
government agencies. 

Co-created PPSR Project Model 

In Co-created projects, members of the public come up with a question or issue, often 
a community concern such as point-source pollution, and then work with scientists 
to answer the question and suggest solutions. Participants are encouraged to take 
part in all stages of the research process, from defining the question to disseminating 
conclusions and asking new questions, and if any part of the project does not work 
well, scientists and public participants work together to modify it for increased 
effectiveness. Thus, public participants exert significant control over Co-created 
projects. Many projects in this category have been labeled “community science” or 
“participatory action research” (Cornwall and Jewkes 1995, Wilderman et al. 2004, 
Fernandez-Gimenez et al. 2008). Examples include some public health or environmental 
restoration initiatives. 
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As methods of informal science education, particularly for helping the public learn about the process 
of scientific investigation, each model has strengths and weaknesses. To better understand the 
characteristics of each model and to learn how PPSR projects can be developed or redesigned to yield 
the greatest possible learning impacts, we have examined a variety of projects of each model type. 

Note that in our attempt to develop models of PPSR we have deliberately excluded public engagement 
in science (PES) activities that involve members of the public in understanding and infiuencing public 
policy as opposed to participating directly in research. PES projects exist worldwide and include 
citizen juries, consensus conferences, and science shops (Rowe and Erewer 2005). Some educators 
might consider PPSR to be one form of PES. This idea is discussed further in the CAISE report Many 
Experts, Many Audiences (McCallie et al. 2009). 
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Part 3. Methodology for Meta-analysis of PPSR Projects 

For our meta-analysis we reviewed ten PPSR projects including examples of each model type — five 
Contributory, three Collaborative, and two Co-Created. Our original plan was to create a purposive 
sample by examining an array of PPSR projects of all types, determining which had been most 
extensively evaluated, and then selecting ten projects for which we could extract project impacts 
from final evaluation reports or other project publications. We quickly discovered, however, that few 
PPSR projects have been evaluated comprehensively, especially those that we have categorized as 
Collaborative or Co-Created. In many cases, evaluations simply have not been conducted. In addition, 
many PPSR projects were started with an emphasis on scientific or community problem-solving goals 
as opposed to educational goals. In other words, we are layering educational models over projects that 
were developed not so much to help people learn about science as to address scientific or community 
needs. In the end we created a convenience sample including most of the USA-based PPSR projects 
that we could locate — either from the NSF ISE portfolio, from the list of projects available at www. 
citizenscience.org, or by reference from colleagues— that included measured and reported outcomes 
or for which we felt that we could infer outcomes from available information. (We excluded from 
analysis most projects developed by the Cornell Lab of Ornithology to avoid oversampling; these are 
included in the Appendix of this report, which lists projects funded by the NSF ISE Program through 
June 2009.) 

To standardize our methods of analysis for each selected project, we developed a rubric based on the 
evaluation framework described in Framework for Evaluating Impacts of Informal Science Education 
Projects (Friedman 2008). We decided to use the Framework for three reasons. First, its categories 
were developed by leading evaluators in the ISE field. Second, all ISE projects funded by NSE are 
now expected to use the Eramework to guide their evaluations, so we seized the opportunity to assess 
its utility in guiding our meta- analysis. Third, we hoped that the rubric would be a first step toward 
developing an organized methodology for comparing outcomes across a variety of PPSR projects, not 
just for this report but for the growing PPSR field in general. 

Our rubric is shown in Table 2. We took the impact categories shown in bold directly from Eriedman: 
Developing understanding and knowledge, enhancing engagement or interest, improving skills, 
changing attitudes, and changing behavior. We also included a sixth category, “other,” which dealt 
primarily with social and community impact. Next, for each of the major categories we developed 
subcategories by reviewing goals and learning objectives for individual PPSR projects. Additional 
subcategories emerged as our analyses were completed, because we discovered that we had not 
anticipated all impact types. 

Eor each project that we reviewed, we examined project descriptions and reports to identify goals, 
objectives, and potential indicators for each impact category. Eor example, a stated goal might be 
increased awareness of biodiversity, and a potential indicator might be changes in awareness based 
on scores on a pre- post-test assessment. Next we determined measured outcomes (which were rare 
for most studies) and inferred outcomes (outcomes that seem to be happening but which are based 
on anecdote or perception as opposed to qualitative or quantitative evaluation). In some cases we 
contacted project staff for additional information. The final reviews were validated across the inquiry 
group by consensus dialogues. 

In most cases, using this rubric to assess project impact was challenging because few if any of the 
projects used anything like it in their original or ongoing evaluations. However, we believe that our 
rubric represents an important step in bringing cohesion to the emerging PPSR field. It can be used not 
only as a guide to project assessment, but also as a tool for project planning and development. 
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Table 2. Assessment Rubric for Describing Impacts of Public Participation in Scientific Research Projects 


Impact category 


Stated goal 


Potential indicators 


Measured outcomes 


Inferred outcomes 


Awareness, knowledge, 
and/or understanding 
(of)^ 










Content (Concepts) 










Process 










Nature of science 










Careers 










Community 












1 

Engagement or interest 
(in)" 










Content (Concepts) 










Process 










Community 










Careers 












Skills ^ 










Asking questions 










Study design 










Data collection 










Data analysis 










Data interpretation 










Discuss results 










Disseminate results 










Using technology 










Writing 










Community 













CO center for advancement of 
C*Cl I OVT informal science education 



insci 

.ORG 



Attitudes '* 










Toward science 










Toward content 










Toward people 










About activities 










Toward species 










About careers 










About theories 










About community 












Behaviors ^ 










Time engaged 










Time outdoors 










Lifestyle changes 










Within community 










Community 

involvement 


■ — — 








Citizen action 










Responsible 
environmental behavior 










New participation 












Other ® 










Social capital 










Community capacity 











Economic impact 
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Impact Categories 

'' Measurable demonstration of assessment of, change in, or exercise of awareness, 
knowledge, understanding of a particular scientific topic, concept, phenomena, theory, or 
careers central to the project. 

Evidence for participant awareness can be observed through direct assessments, self-reports, or 
self-reflection. 

^ Measurable demonstration of assessment of, change in, or exercise of engagement/interest 
in a particular scientific topic, concept, phenomena, theory, or careers central to the project. 

Indicators of engagement and interest may include length of commitment to, or depth of involvement 
with, a project or choices to further pursue content knowledge or related activities beyond a project’s 
scope. 

^ Measurable demonstration of the development and/or reinforcement of skills, either 
entirely new ones or the reinforcement, even practice, of developing skills. These tend to be 
procedural aspects of knowing, as opposed to the more declarative aspects of knowledge 
impacts. Although they can sometimes manifest as engagement, typically observed skills 
include a level of depth and skill such as engaging in scientific inquiry skills (observing, 
classifying, exploring, questioning, predicting, or experimenting), as well as developing/ 
practicing very specific skills related to the use of scientific instruments and devices (e.g. 
using microscopes or telescopes successfully). 

Indicators of skill development could include a demonstrated degree of proficiency (such as the 
ability to identify species) or adoption and employment of a science-related skill (for example, 
prediction, argumentation, or synthesis). 

^Measurable demonstration of assessment of, change in, or exercise of attitude toward a 
particular scientific topic, concept, phenomena, theory, or careers central to the project or 
one’s capabilities relative to these areas. Although similar to awareness/interest/engagement, 
attitudes refer to changes in relatively stable, more intractable constructs such as empathy 
for animals and their habitats, appreciation for the role of scientists in society or attitudes 
toward stem cell research 

Some standardized metrics for evaluating attitudes are available, but many rely on participant self- 
reports. In the context of PPSR, outcomes may include changing participants’ attitudes about the role 
of science in their lives or communities, or improving attitudes about particular species, resources, or 
partnering organizations. 

^Measurable demonstration of assessment of, change in, or exercise of behavior related 
to a STEM topic. These types of impacts are particularly relevant to projects that are 
environmental in nature or have some kind of a health science focus since action is a desired 
outcome. 

Evidence of behavior change might include participants’ self-reported intentions, and longitudinal 
tracking to determine whether such behavior change has occurred. 

® Other 

The NSF framework allows for measuring outcomes that do not fit well into the other five categories. 
We chose to look for evidence that projects affected participants’ views of their communities, 
both socially and economically, and/or participants’ interactions with other people and scientific 
institutions. 
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Part 4. Public Participation in Scientific Research Sample Project 
Descriptions 

Contributory Projects 

The Birdhouse Network 

The Birdhouse Network (TBN), developed and 
operated by the Cornell Lab of Ornithology (CLO) 
with funding from the National Science Foundation, 
operated from 1996 until it was absorbed into 
NestWatch in 2007 (www.nestwatch.org). For 
TBN, participants placed nest boxes in their yards 
or neighborhoods and then collected data about the 
birds that bred in them during spring and summer. 

The data were submitted to CLO where scientists 
interpreted the information to develop a better 
understanding of temporal and geographic variations 
in the reproductive strategies of cavity-nesting birds 
throughout the United States and Canada. The project 
also aimed to enhance volunteer appreciation and 
understanding of the biology and ecology of breeding 
birds along with an understanding of the process by 
which scientific research is conducted. 

During the project’s 10-year run, approximately 5,000 
participants monitored 75,000 nests in 49 states and 
seven Canadian provinces. Typical participants were 
older, well educated, retired females. Participants 
were encouraged to monitor nest boxes once or twice 
per week during the breeding season. Some observed 
just one or two boxes per week, while others 
monitored several dozen or even hundreds of boxes. 

The average number of boxes monitored was 22. 

This project included a comprehensive website housing all project materials — instructions, data forms, 
and resource guides. Project staff also communicated with participants via a semiannual printed 
newsletter called “Inside the Birdhouse,” email updates throughout the year, individual emails and 
phone calls, two email discussion listservs, and Birdscope, CLO’s quarterly newsletter. Participants 
learned how to conduct the project on their own either by reading through a printed Research Kit (in 
the early years of the project) or by reading the directions online (later in the project when the Internet 
had become widely established as an easy means of communication). 

Participants in TBN conducted repeat visits to nest boxes, recording individual nest locations, site 
characteristics, species using the box, and number of eggs and young in the nest. They also recorded 
important dates such as estimated first egg date, hatch date, and fledge date for each nest. After 2001 
all data were entered online, where the information could be viewed and edited throughout the year. 
While participants did not have access to the raw data, they could visualize data through a variety 
of “canned” data queries. (Participants in newer CLO projects, such as eBird and NestWatch, have 
access to raw data.) Some participants provided suggestions to project staff — for example, interesting 
questions for scientists to study or new data collection methods — but this was neither required 
nor expected. In one “substudy,” which focused on blowfly parasitism, participants collected and 
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Seeing young birds up close brought joy and 
discovery to participants in The Birdhouse Network. 
Photo courtesy The Birdhouse Network 




submitted complete nests to a partnering researcher (federal permits needed to be obtained to allow 
this activity). 

To date, scientists have used TBN data to examine various breeding parameters and have published 
results in 11 peer-reviewed scientific journals. 

Evaluation of TBN included a mix of qualitative and quantitative research methods. In 1997, 
participants were sent a pre-test survey before receiving educational materials and a post-test survey at 
the end of the field season. These surveys attempted to measure changes in science content knowledge 
(in this case, biology of breeding birds), science inquiry knowledge, and attitudes toward science 
and the environment. Comparisons of pre- and post-project surveys showed that TBN participants 
significantly increased their knowledge of bird biology (Brossard et al. 2005). 

To determine whether participants had increased their knowledge of science process as a result of 
participating in TBN, evaluators gathered self-reports via a close-ended question followed by an 
open-ended question to assess the self-report responses. In general, participants did not seem to 
increase their science process knowledge, and three factors could have contributed to this finding. 
First, most participants probably joined TBN to learn about cavity-nesting birds as opposed to 
the process of science. Second, the Research Kits contained little information explicitly aimed at 
describing or explaining the process of science and showing participants how their activities fit into 
that process. Third, the project procedures did not ask participants to reflect on the scientific process 
(Brossard et al. 2005). However, interviews with project stalT show that some participants did ask 
original questions and that some of these questions were refined and developed into new TBN studies. 
Also, some participants set up their own experiments and shared results with email discussion groups 
and CEO scientists — at least a dozen participant-initiated studies are currently presented on TBN’s 
website (now located at www.nestinginfo.org). These results suggest that Contributory PPSR projects 
have the potential to incorporate multiple facets of scientific inquiry. 

To assess participant attitudes toward science, the evaluators used a modified version of an existing 
standardized instrument — the Attitude Toward Organized Science Scale (ATOSS) (National Science 
Board 1996). Results showed little to no change as a result of TBN participation. To assess changes in 
environmental attitudes, evaluators used a version of the New Environmental Paradigm (Dunlap and 
Van Liere 1978), which revealed no evidence that project participation led to a less anthropocentric 
worldview of nature. The fact that TBN participants entered the project with very positive attitudes 
toward science and the environment compared with the “average public” could have yielded these 
findings (Brossard et al. 2005). 

For considering potential behavior changes among participants, anecdotal evidence is available from 
email correspondence and online discussion groups, examination of products produced by TBN 
participants, and participant interviews (Phillips et al. 2006). This information shows that some 
participants increased “pro-bird” behaviors such as erecting nest boxes, providing water sources, 
planting native plants, designating yards as “wildlife sanctuaries,” and disseminating information to 
other community members. 

Spotting the Weedy Invasives 

Spotting the Weedy Invasives, operated by Rutgers University and the New York/New Jersey 
Trail Conference and sponsored by the United States Department of Agriculture, engages hikers 
in identifying and mapping invasive plants along 160 km of trails in the New York/New Jersey 
Highlands (www.trails.rutgers.edu). The project began in 2006 with four researcher-defined goals: 

1) Generate a long-term dataset about the distribution and abundance of key invasive plant species 

2) Help participants learn about the ecology of invasive plants 
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3) Help participants understand how project data will be used to draw conclusions about the 
establishment and spread of weedy invasive plants 

4) Encourage participants to take personal action toward reduction or eradication of invasive 
plants. 

Each year the project attracts from 30 to 40 participants, 
most of whom are members of the NY/NJ Trail conference. 

About half of participants are men and half women; the 
average age is mid 50s; and about 85% of participants are 
currently employed. 

Participants spend at least 30 hours in the field each year. 

At the beginning of each season they attend a one-day 
training session at which they learn how to recognize 
the target species and collect required data. Additional 
training material is available online. After training, pairs 
of volunteers are assigned to hike a two-mile stretch of 
trail, stopping every 0.3 km to record the presence and 
relative density (low, medium, or high) of a selected suite 
of invasive plant species (averaging about 12 species 
depending on the year). Data validation efforts include 

1) volunteers submitting a plant clipping of the first 
plant spotting 

2) all volunteers walking a common trail for which 
invasive plant distribution is known 

3) 25-50% (depending on the year) of data collected 
being validated by an external expert. 

While participation in this project is limited primarily to the collection of data, during the training 
session participants also practice the development of explanations. They hear about how conclusions 
are drawn, and they are offered the opportunity to help disseminate conclusions about the study to 
local groups and/or land managers. 

Project staff have investigated several aspects of project impact via written questionnaires developed 
by project staff and administered at three points during the project: 

1) After the initial training session; 

2) After a debriefing session held immediately following data collection; and 

3) Six months after the data collection period. 

Note that of the results discussed below, 1) and 2) are the sources of pre-test data and 3) is the source 
of posttest data. Most participants filled out questionnaires during the initial training and the debriefing 
sessions, while 35% returned the 6-month follow-up survey. All data were self-reported information 
(project director, personal communication). 

All individuals indicated that they enjoyed project participation and felt they had gained knowledge 
about invasive plants. They also enjoyed the notion of doing something positive for science. Nearly 
three-quarters of respondents reported that they significantly increased their content knowledge, and 
pre- and posttest comparisons showed a 20% increase in the number of individuals who were able to 
provide correct responses to questions probing general background information about invasive plants. 
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Volunteers for Spotting the Weedy Invasives 
attend training sessions before they begin 
to collect data. Photo courtesy Spotting the 
Weedy Invasives 



Participants who completed pretests generally had a sense that science can be uncertain and that 
conclusions are tentative. They tended to rank highly the extent to which evidence should be visible, 
thus underappreciating the role of inference in drawing conclusions. Slightly over half of respondents 
articulated the necessity of controlling variables during an experimental process. However, fewer 
than one-third were able to adequately tease causation apart from correlation. Understanding of these 
issues did not measurably change between pre- and post-project participation, but views of science 
did change for some individuals. Nine of the 33 respondents stated that they possessed a greater 
understanding of the scientific process. Their responses varied in terms of sophistication — some 
held the notion that science is a way of reasoning, while others saw science only as experimentation 
and observation, most often through trial and error. Two individuals stated that science equates 
to a content area such as biology or physics. Variation in these responses did not correspond to 
professional background. 

With respect to general environmental literacy as defined by the North American Association for 
Environmental Education (2000), individuals came to the project greatly concerned about the 
environment, and their concern did not change measurably following participation. However, 
respondents perceived their environmental knowledge to be only moderate (both pre- and post- 
participation), indicating that concern about environmental issues is not necessarily matched by the 
perception of knowledge about those issues. Civic awareness, as defined by awareness of the ways in 
which environmental problems are solved, was moderate pre-participation and increased slightly after 
participation. Participation did not affect locus of control (i.e., sense of empowerment resting within 
the individual versus beyond individual control). 

Some slight changes in behavior were noted among respondents. When participants began the 
project, 78% said that they considered invasive plants when making plant purchases, and 30% viewed 
invasive plants and other environmental issues as infiuential in their voting decisions. Although the 
latter measure did not change, the proportion of people considering invasive plants while making 
purchases increased to 86% at project completion. In addition, 39% stated they had begun to take 
notice of invasive plants in the environment, and 43% reported that they talked to others about 
invasive plants. Planting habits were changed by 9% of respondents, and 6% went on to seek other 
volunteering opportunities. Common explanations for individuals who did not take action included 
time constraints and a sense of futility given the scale of the problem. 

ALLARM Add Rain Monitoring Project 

The Alliance for Aquatic Resources Monitoring (ALLARM) Acid Rain Project (www.dickinson. 
edu/allarm) operated throughout the state of Pennsylvania from 1984 through 2005. It was started 
as a data-gathering and informal science education project by Candie Wilderman, Professor 
of Environmental Science at Dickinson College in Carlisle, in response to an observation by 
Pennsylvania state legislator John Broujos that, although Pennsylvania receives the most acidic 
deposition of any state in the nation, Pennsylvania citizens are mostly unaware of the problem. For 
this project, trained participants made weekly visits to a stream of their choice where they tested the 
water for pH and alkalinity. 

The project had many researcher-defined goals. Scientifically, the project aimed to document patterns 
of response to acid deposition in Pennsylvania streams. In the area of education, the project aimed 
to help participants gain knowledge of the extent and impact of acid deposition on streams and to 
develop awareness of the variability in aquatic systems and the challenges involved in characterizing 
them. In the realm of conservation, the project desired to help participants first build a strong sense 
of place by visiting a single site on a weekly basis and then to build a sense of stewardship for that 
place. Finally, the project aimed to empower participants with first-hand knowledge of the responses 
of streams to acid rain so that they could, if they chose, participate in community efforts to find 
solutions. 
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Each year the project involved about 150 adult participants with the 
following audience composition: 75% male; 72% college educated; 

63% small town/rural, 30% suburban, 7% urban; and 38% retired. 

As in most water quality monitoring projects, this one required 
volunteers to demonstrate their competency in collecting and 
analyzing samples before participating. They began the project by 
attending one or two training workshops at which they learned how 
to analyze water for pH and alkalinity and to choose study sites. 

After the workshop they sent in a split sample for Quality Control/ 

Quality Assurance (QC/QA) testing. (In this procedure, the volunteer 
splits a sample into two, measures his/her half, then sends the other 
half to a lab for testing and comparison to the volunteer’s results.) 

Individuals who passed this assessment chose a stream site or sites, 
usually near their home and complying with site-selection criteria; 
visited their site(s) once per week; tested the stream for pH and 
alkalinity using a field kit developed by ALLARM; and filled out a 
data form. Participants averaged about two hours of effort each week 
and needed to pass QC/QA once per year to continue to contribute 
data. Although individuals networked with other participants if 
they chose to attend additional training workshops, they worked 
independently for the bulk of the project. 

Project data were managed and analyzed by ALLARM staff but were available to the public through 
the Pennsylvania Spatial Data Analysis website, the ALLARM website, and the Pennsylvania 
Keystone Monitoring Database. Participants received annual reports including analyses of the 
patterns found at their stream site(s) within the larger context of all sites sampled. Participants also 
had opportunities to attend presentations on the science and policy of acidic deposition presented 
by ALL ARM staff on a roughly bi-monthly basis. ALLARM staff also disseminated results through 
scientific papers, conference presentations, legislative testimony, newsletters, and articles (for 
example, Wilderman and Schott 1991; Wilderman and Vorhees 1999; ALLARM 1986-2009). 

While we have characterized the basic design of this project as contributory, the project did include 
the opportunity for participants to go beyond data collection if desired. Lor example, participants often 
gathered local information on their stream sites to help explain detected water quality patterns. Also, 
many participants chose to disseminate project conclusions via newspaper editorials, contact with local 
and state officials, and contact with angler and conservation clubs, even though ALLARM offered no 
training or formal support for participants to engage in these activities. 

Linally, some participants discussed project results and asked new questions at training workshops or 
at presentations on the science and policy of acid deposition. In fact, because so many participants 
began asking questions that went beyond the single issue of acid deposition, ALLARM expanded 
its focus to include a wider range of water quality issues, changed its name from Alliance for Acid 
Rain Monitoring to Alliance for Aquatic Resources Monitoring, and broadened its project design 
from Contributory to a Co-created model (Wilderman et al. 2003). So, although the “final” step in the 
scientific process was not a formal step in the operational process of the original project, the discussing 
of results and the asking of new questions happened to such an extent that the project itself was 
significantly expanded to address a broader range of community-based issues. 

The ALLARM Acid Rain Project received informal evaluation by project staff. Assessments included 
feedback through interviews, discussions at workshops, participant surveys, workshop evaluations, 
quality control passage rates, and examination of participant-created products emerging from the 
project. 




ALLARM acid rain monitoring 
volunteers follow set protocols 
to test water pH. Photo courtesy 
ALLARM 



■! center for advancement of 
v*Cl I OvT informal science education 



insci 

.ORG 



ALLARM staff noted numerous projeet impacts. Many participants wrote increasingly sophisticated 
comments on data forms, indicating that they were gaining knowledge of many concepts about acid 
deposition and stream quality. By participating in the QA/QC, participants gained an appreciation 
of the importance of validation in the scientific process. And some participants clearly increased 
their understanding of the concepts of variability and sampling intensity and used this knowledge 
to describe the acid deposition problem through letters to the editor, discussions with government 
representatives, and presentations at community events. 

Participants were deeply engaged with this project. About 90% of those who started monitoring stayed 
with the project for the required one-year period and about 20% stayed from 2-5 years. Also, about 
5% of volunteers were observed to take direct action as a result of their monitoring. Some participants 
reported feeling isolated in the project, and although training workshops helped to develop a sense of 
community, individuals who attended workshops did not seem to engage in continued networking. 

Participants clearly developed skills in data collection. About 95% of all volunteers eventually 
passed QC/QA tests and went on to collect weekly data. However, the project did not seem to help 
participants develop other skills. For instance while volunteers generally understood the criteria for 
site location and were able to choose suitable study sites, they had little understanding of the overall 
statewide study design. There was also a great deal of variability in the level of understanding of the 
data among volunteers, and few volunteers took it upon themselves to independently disseminate the 
analyses that ALL ARM presented to them. However, project staff occasionally received copies of 
letters written and anecdotal information on conversations and presentations given by participants. 

Overall attitudes toward science did not seem to shift as a result of project participation. Most 
participants felt that they learned a particular skill to help them contribute to the work of scientists, but 
that their role was that of technician. However, participants consistently reported enjoying workshops 
and monitoring activities. Most had the sense that the activities were making a useful contribution to 
the scientific database, largely because they trusted the scientists involved. 

While project staff had no knowledge of direct citizen action resulting from the project, volunteer 
interest in expanding the project beyond the issue of acid deposition into broader environmental 
monitoring ultimately led to the expansion of ALLARM into new watershed-based projects. Some 
of the acid rain volunteers participated in the broader program of watershed monitoring when it was 
created. 

Monarch Larva Monitoring Project 

The Monarch Larva Monitoring Project (MLMP) was begun by researchers at the University of 
Minnesota in 1997 to involve citizens in collecting data to help explain the distribution and abundance 
patterns of monarch butterflies during the breeding season in North America and to inform monarch 
conservation (www.mlmp.org). Sponsored by the National Science Foundation from 2002 through 
2005, MLMP seeks not only to gather important data but also to provide citizens with hands-on 
experience in scientific research, thereby enhancing their appreciation and understanding of monarchs, 
monarch habitat, and the scientific process in general. 

The audience comprises all ages and demographics. About half of participants, including teachers, 
naturalists, and parents, report that they monitor monarchs with three to five children each. While the 
project has not collected detailed data on the total number of participants — only one volunteer records 
data for each site — participants have now monitored 956 field sites in 28 U.S. states and 3 Canadian 
provinces, so the total number of participants is likely in the thousands. 

Participants learn project procedures either by reading directions online or, if they live near a 
participating nature center, by attending an in-person training session at which they learn about 
monarch biology, practice monitoring and data-entry protocols, and receive all needed materials. 
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Once trained, volunteers ehoose and deseribe their own monitoring 
sites, which include backyard gardens, abandoned fields, pastures, 
and restored prairies located throughout the monareh’s breeding range 
(mainly the eastern half of the United States and southeastern Canada, 
although some volunteers monitor a western population in coastal 
California). The only requirement is that the sites contain milkweed 
{Asdepias spp.), the monarch’s larval host plant. 

Once their sites are ehosen, partieipants conduet weekly surveys of 
monarchs and milkweeds. They randomly choose as many plants as 
possible to census (typically from 50 to 500), then record the number 
of milkweeds examined and the numbers of eggs and larvae observed, 
identifying larvae to a specifie instar (monarchs go through five larval 
instars). They enter their data into an online database, whieh they 
can modify throughout the season. At the end of the season they send 
projeet managers hard eopies of their data sheets, whieh are used to 
spot-cheek the online data and then archived. Summaries of the data 
are made available online and through an annual newsletter. Raw data 
from individual sites can be downloaded from the website, and other 
data are made available upon request. Many people, including both 
participants and non-participants, request raw data for informational, 
scientific, and educational purposes. No data analysis tools are provided, however. 

Optional data-collection aetivities inelude assessing milkweed quality, measuring rates of parasitism 
by parasitoid wasps and fiies, and collecting weather data. The assessment of milkweed quality 
involves comparing plants occupied by monarchs to random plants by measuring the same 
characteristics of each (height, reproduetive status, age, herbivore damage, and the presence or 
absence of invertebrates). Partieipants who engage in the parasitism study colleet monarch eggs or 
larvae (mainly fourth or fifth instars), rear the eaterpillars indoors, record whether they survive to 
adulthood and, if they do not, record what causes their death. 

MLMP staff have colleeted numerous comments that show the importance and significance of the 
projeet for its participants. For example, a high school student from Minnesota stated, “ft is amazing 
to me that when people all over the country take a little time every week, and even more in some 
cases, to count butterfly eggs, the end result is a network of data that can help us deeipher where the 
butterflies go and when and how . . . This is real life proof that when everybody works together, things 
can be done.” And another student stated, “MLMP has given me the opportunity to learn how a truly 
massive research project functions, but has also let me understand how important every piece of data 
is, no matter how small, when needed to reach a conclusion.” Still another student states, “The MLMP 
has helped me become a better scientist in so many ways. Most importantly, it gave me a large interest 
in seience. ft encouraged me to ask questions such as why and how and to find these answers through 
experimenting.” 

MLMP also has been assessed through an evaluation that focused on understanding the impacts of the 
projeet on children who were being mentored by an adult participant (Kountoupes and Oberhauser 
2008). The most commonly noted outeome, ascertained through detailed interviews with MLMP 
participants, is learning and understanding about real scientific research. Adults perceived that children 
felt like real scientists and were proud of their contributions. 

Interview data also highlighted the importanee of the soeial aspects of the program. The shared 
experience allowed children to meet new friends with like interests while enjoying time together in the 
outdoors. One interviewee described the experienee for her group of ehildren as “seience bonding.” 




A volunteer monitors monarch 
butterfly populations in Duluth, 
Minnesota. Photo courtesy 
MLMP 
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Another emphasized a need for alternatives to sports-eentered reereation, describing childrens’ thirst 
for learning about nature. 



Although this project has focused primarily on data collection, 
many participants, especially teachers and naturalists, have 
chosen to engage youth participants in a variety of related 
science-process activities of their own design, including 
defining new questions for study, developing hypotheses, 
designing data-collection methodologies, interpreting data 
and drawing conclusions, and disseminating results. Not 
surprisingly, adults who engage youth in these activities report 
the strongest learning gains. For this reason, the MLMP is 
expanding the program by supporting volunteers who use their 
MLMP experiences to conduct independent investigations. In 
one case, an independent investigation instigated by a volunteer 
led to a paper co-authored by three volunteers and published 
in the peer-reviewed literature (Oberhauser et al. 2007). Other 
projects are highlighted in the annual newsletter, and project 
coordinators correspond frequently with volunteers who are 
conducting their own studies. 

Community Collaborative Rain, Hail, & Snow Network 

CoCoRaHS (the Community Collaborative Rain, Hail, and Snow Network) is a community-based 
network of volunteers of all ages and backgrounds working together to measure and map precipitation 
(rain, hail, and snow) (www.cocorahs.org). Begun at the Colorado Climate Center at Colorado 
State University in 1998, CoCoRaHS is now sponsored by the National Oceanic and Atmospheric 
Administration (NOAA) and a variety of other organizations. Anyone interested in watching and 
reporting weather conditions is invited to participate, and more than 12,000 observers are currently 
contributing observations from 39 states. 

Goals of CoCoRaHS include providing large quantities of high-quality precipitation data for a 
variety of users while encouraging citizens to have fun participating in meteorological science and 
heightening their awareness about weather. From 2003 to 2006 the project was supported by the 
National Science Foundation to help the organization provide enrichment activities in water and 
weather resources. 

Using a simple rain gauge, CoCoRaHS volunteers take measurements of precipitation from as 
many locations as possible each time a rain, hail, or snowstorm crosses their area. They record their 
reports on the project website where the data are organized, displayed, and analyzed by a variety 
of individuals and organizations including the National Weather Service, hydrologists, emergency 
managers, city utilities, farmers, and even insurance adjustors. Training is provided either online via 
downloadable slide shows or, in some states, at local training sessions. Feedback to volunteers is 
provided primarily through a bi-monthly newsletter that can be received by email or downloaded as 
desired. 

At the end of the grant period, project staff had received approximately 7,000 volunteer applications, 
and nearly 4,900 participants, located primarily in the Midwest, collected data at some point during 
the project. Most of those who applied were adults in the age range from 55 to 80 years old, with a 
secondary peak of applicants, representing families with school-aged children, of parents in their 30s 
and 40s. Approximately 60% of registrants were male and about 40% female. 

The duration of participation varied widely. Some participated sporadically or for a short period of 
time, while others participated thoroughly and continuously after first getting started. Some reported 
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every day, while others reported only during storms. Those who 
signed up but did not participate offered a wide range of reasons 
ranging from expense of rain gauges to lack of time for attending 
training or collecting data. 

Project staff found that the most effective training, especially for 
beginners new to data collection, was face-to-face workshops. 

Individuals who attended group training sessions were more likely 
to participate and to participate longer. However, providing personal 
training for all participants is difficult for large-scale projects, and the 
project staff feel that successfully growing the project will depend 
largely on increasing training options. 

Evaluation of this project included participant inventory and tracking, 
precipitation data compilation (providing information on the duration 
and spatial distribution as well as data quality provided by CoCoRaHS 
volunteers), participant surveys, and focused interviews with a 
randomly selected group of several dozen participants (CoCoRaHS 
2007). 

Project staff identified several project activities during which 
learning could take place, including siting and installing rain gauges, 
measuring and reporting precipitation, examining self-collected data 
over time, and reading results from newsletters. However, little information about learning seems to 
have been collected for project evaluation. The only information available from the final project is 
the statement “from surveys and interviews, most participants are able to define specific learning that 
they have identified. In many cases, the learning is simply an increased awareness of the surrounding 
environment and how it is affected by precipitation. The most common learning is that precipitation is 
much more variable than most people realized.” 

Collaborative Projects 

Salal Harvest Sustainability Study 

The Salal Harvest Sustainability Study, sponsored by the Community Forestry and Environmental 
Research Partnership through a researcher (Heidi Ballard) at UC Berkeley and the Northwest 
Research and Harvester Association (NRHA), ran from 2001 through 2004. The project took place 
on Washington’s Olympic Peninsula and involved harvesters of salal {Gaultheria shallon) in research 
to determine the effects of harvest intensity on the plant’s growth and biological and commercial 
production. Salal is an evergreen shrub harvested as a non-timber forest product (NTFP) for use as 
greenery in the fioral industry. The increasing harvest rate in recent years is causing concern about the 
impacts on habitat by some public land managers. 

Specific goals of the study were to determine effects of different harvest intensities to inform 
management of the plant and improve harvesting practices; to involve and potentially empower salal 
harvesters in research affecting their livelihoods; and to provide harvesters with forestry and technical 
job skills that could help them gain employment in forest management. Approximately 35 salal 
harvesters took part in the project, about 10 of whom participated during all four years. Most were 
undocumented Latinos between the ages of 20 and 40 who were learning English. A few were Asians 
or Caucasians. All participants depended at least part time on harvesting salal for their livelihood; for 
most it was a full-time job. 

The project began with the researcher training the 10 harvesters onsite, either individually or in small 
groups of three to four participants. They learned about site selection, methods design, data collection, 
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Participants in CoCoRaHS 
measure amounts of rain, hail, 
and snow. Photo by Henry 
Reges, CoCoRaHS 




and interpretation of results. Then, over the eourse of the year, eaeh participant spent about 40 hours 
working on site assessments, research design, and data collection. During the second and third years 
harvesters spent about 20 hours per year collecting data. Also during the third year, all 10 participants, 
along with about 25 additional harvesters, attended a workshop to learn how to interpret graphs and 
draw conclusions from evidence and observations. During the fourth year each participant spent about 
three hours discussing and disseminating project results. The project scientist compiled the data and 
kept in touch with participants either in person during the field season 
(May-November) or by phone. 

This project is considered collaborative because, while the project 
scientist had the final say on defining the research questions and 
developing study hypotheses, participants played a key role in each. 

Despite their relatively short time living and working in the forests of 
the Olympic Peninsula (12 years or less), the harvesters had developed 
local ecological knowledge that often surpassed that of forest 
professionals, including knowledge about ways in which understory 
plants respond to timber management practices such as thinning, and 
understanding of which overstory conditions help commercial-quality 
salal to grow best. They also knew the various harvest practices, 
and helped design the project to test the two practices most often 
used. They chose research locations that refiected the variety of 
environmental conditions in the area, such as differing elevations 
and forest stand types. The researcher and harvesters collaborated to 
design response variables in the plant’s growth which would detect 
and measure specific impacts due to harvesting, including percentage 
of leaves damaged by insects or fungi and stem density of harvestable 
new shoots, variables not found in the literature. Harvesters collected all types of project data and 
applied all harvest treatments, calibrating treatments across individuals and sites. 

After treatment, the researcher took responsibility for analyzing the information. However, the 
harvesters played a key role in interpreting the data at a large workshop facilitated by the researcher 
and leaders of the NRHA, during which harvesters interpreted graphs of harvest yield results. They 
analyzed why some results differed from their hypotheses, why sites responded differently to the same 
harvest treatments, and how results could be used for management recommendations. Conclusions 
were summarized by the researcher and disseminated to the local public, private land managers, 
Washington State University Cooperative Extension, and the NRHA. 

Impacts of this project were assessed in two ways. First, the researcher used semi-structured 
interviews to collect ethnographic data and information about participants’ knowledge of local 
ecology throughout the course of the project. In addition, an outside evaluator interviewed seven 
participants during the final year of the study to determine what they had learned during the project, 
what they thought the researcher learned from them, and their recommendations for other scientists 
who would like to work with similar participants in the future (Ballard and Huntsinger 2006; Ballard 
2008; Collins et al. 2008). 

Based on information collected during the interviews, harvesters appeared to develop awareness 
and understanding of science concepts, such as understanding of plant associations, ecological 
relationships, and forest succession. In addition, in their discussions of experimental design, data 
collection, and results interpretation, harvesters demonstrated substantial awareness of how science is 
conducted. While three harvesters expressed frustration with the tedious detail of the scientific process 
and felt it a waste of time, four others expressed interest in working with other scientists again, 
especially to study the effects of fertilizer on commercial production and how white pine blister rust 
could be treated using pruning methods for commercial use of the pine boughs. Both of these studies 
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Participants in the Salal Harvest 
Sustainability study depend on 
salal for their livelihood. Photo 
by Heidi Baiiard 



were pursued in collaboration with Washington State University Extension scientists during following 
years, providing evidence that these types of programs can generate novel research questions. Two 
harvesters inquired about USDA Forest Service Technician jobs, reflecting a desire to move toward a 
job requiring higher skill, although they did not apply owing to their legal status. 

The project also increased the ability of participants to collect held data, to record and observe 
consistently, and to use measurement instruments. Participants in the results interpretation workshop 
also gained skills in reading and interpreting graphs, drawing conclusions from evidence, and 
explaining how the results compared to their own observations in the forest. 

Some harvesters expressed increased concern for the environment, for the health of the forest, and for 
the sustainability of the resource. Several expressed a new awareness of how their harvesting practices 
affected the health of the forest, although they stated that economic pressures to harvest would heavily 
outweigh environmental concern. 

Importantly, harvesters who participated in the research project demonstrated an increase in 
involvement in the NRHA and in the harvester community as a whole. Harvesting is an isolating 
livelihood, especially for illegal workers, so any group participation vastly increases social capital. 
Project participation also led to increased harvester credibility and improved relationships between 
harvesters and forest managers. Rather than being perceived either as invisible or as a nuisance, forest 
managers began to see harvesters as holders of important ecological knowledge and skills and started 
to view their contributions as valuable (Ballard 2008). 

Community Health Effects of Industrial Hog Operations 

Community Health Effects of Industrial Hog Operations (CHEIHO) was a project conducted 
by a partnership between the Concerned Citizens of Tillery (CCT) and academic researchers at 
the University of North Carolina, Chapel Hill (UNC-Chapel Hill) School of Public Health from 
2003-2005. The CCT is an active community organization located in a rural and predominately 
African-American town in North Carolina. CHEIHO was an epidemiological study of the acute human 
health effects of pollution caused by industrial hog operations located near residences (www.cira-unc. 
org/collabs). Because high-density swine production often occurs in communities comprising low- 
income residents and/or people of color, the project focused on issues of environmental justice as well 
as public health. In earlier research projects that quantifled the disproportionate location of industrial 
hog farms in poor communities, community members participated in evaluating data quality through 
their knowledge of local hog operations, making decisions about how to define study populations, 
choosing and defining variables for analysis, and interpreting results. Hence, the CHEIHO project built 
on other PPSR environmental health research projects that the CCT and the university have conducted 
since 1998 (Farquhar and Wing 2003, Wing et al. 2008). 

The goals and study questions for this and the other projects of the CCT-UNC-Chapel Hill partnership 
originated in the exposed communities. Specific goals of the CHEIHO project were to integrate 
ethnographic and epidemiological research on acute exposures and their relationships to both 
health-related outcomes and quality-of-life concerns; to provide environmental health education 
to community members; and to promote community participation in environmental and social 
justice movements. Data were collected from September 2003 through September 2005 in 16 rural 
communities in eastern North Carolina. The study included 104 volunteers, primarily African 
Americans, who were non-smokers over the age of 18 living within 1.5 miles of at least one industrial 
hog operation. Sixteen of these participants recorded data from air pollution monitoring equipment 
once per week in each community for two weeks. The remaining 88 participants collected self- 
administered health data twice each day for 14 days. All participants underwent a three-hour training 
session at which they practiced Ailing out a data collection diary and built rapport with each other and 
the project research team. Each volunteer also participated in six hours of interviews. 
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The scientific team included a core of three 
community leaders affiliated with CCT as well 
as three university researchers. These core 
participants conceived of the research question 
based on problems they experienced in their 
community, designed the study, conducted the 
training sessions, provided data-quality check 
visits to volunteers once during their two-week 
data-collection sessions, helped to analyze 
the data, and facilitated numerous community 
meetings. The core team also published project 
data in public health journals, and CCT used the 
research results to lobby for change in land use 
policies in the region. 



The research questions included: What was the 
frequency, magnitude, and duration of swine 
odors experienced by project volunteers? What 
were the levels of several airborne toxins in 
the area around hog operations, how were 
they related to odors experienced, and how 
were pollution levels related to acute health 
effects experienced by residents (such as lung function, blood pressure, mood, stress, and quality of 
life)? How were the cultural and social contexts of rural life related to experiences of environmental 
exposures and quality of life? 



Goals and questions for the Community Health Effects of 
Industrial Hog Operations project originated in the affected 
community. Photo by Gary R. Grant 



The 88 participants who were involved primarily in data collection recorded observations about 
hog odor outside their residence at predetermined times; recorded whether they experienced any 
irritation of the eyes, nose, throat, or had any physical health symptoms; took their own blood 
pressure; collected a sample of saliva; and blew into a lung function monitor. All these participants 
also discussed summaries of the data analysis during community meetings. Finally, the core research 
team and many of the additional participants were involved in disseminating project conclusions by 
presenting at community meetings in surrounding areas. 



Information about the impacts of this project on land use policy, environmental justice research, and 
community health education have been reported in Wing et al. (2008). These authors did not, however, 
formally measure specific knowledge about science content or process. However, project personnel 
believe that training in the data collection procedure resulted in a significant increase in awareness, 
knowledge, and understanding of human health and air quality issues as well as in understanding of 
data collection methodologies including making careful observations, looking for patterns in data, and 
being consistent in data collection. They also note that “participants gained confidence and a greater 
sense of legitimacy by seeing their experiences and views embedded in a scientific process in which 
they participated.” This research involved members of the public who felt marginalized and placed the 
experiences of these individuals in the context of a scientific study, empowering communities through 
connections with professional research. 

Participants in this project also made new connections with neighbors and organizations, increasing 
social capital. Wing et al. (2008) report that participation in research broadened some participants’ 
perspectives on community involvement and environmental injustice and awakened awareness about 
the potential for change. For example, during the qualitative interview following the quantitative data 
collection, one participant stated “I never thought about it [the concentration of hog houses] . . .The 
day after we did the class [CHEIHO training], I rode around and counted how many hog houses. I 
counted 22. I never knew there were that many hog houses . . . [or] how in the world they got here.” 



• 1 • • 

center for advancement of 35 1 

v*CllOv7 informal science education 





While the CHEIHO project did not specifically measure science education impacts, community 
education about environmental health was an explicit goal and was documented during the project 
(Wing et al. 2008). An additional goal was to build the organizing and communication skills of 
community members to take action in local, regional, and state arenas to address the environmental 
health and injustice problems in their community. In this light, the authors state that evidence of 
their education goals being met can be found in the community action that led to policy change and 
improved health. For example, the work of the Concerned Citizens of Tillery and UNC Chapel Hill 
researchers has resulted in an ordinance regulating the corporate hog industry. 

Invasive Plant Atlas of New England 

The Invasive Plant Atlas of New England (IPANE), initially funded by the U.S. Department of 
Agriculture, is another project that seeks to engage volunteers in identifying and mapping invasive 
plants (www.ipane.org). The project was begun in 2001 to develop a widespread early detection 
system. Since that time, project organizers developed educational goals that called for increasing 
content knowledge and science process skills among participants. These goals include not only the 
understanding of the effects of invasive plant species on native ecosystems but also an understanding 
of how invasive species data are collected. The project also aims to engage participants in management 
actions through partnerships with management organizations. 

So far the project has trained 900 individuals, all adults, and now includes approximately 500 active 
volunteers. Participants are recruited through fiyers and project partners, including the University of 
Connecticut, the Silvio O. Conte Refuge of the U.S. Fish & Wildlife Service, and the New England 
Wildfiower Society. 

Participants begin by taking part in a one-day training session at 
which they learn how to identity and collect data on invasive plants 
and how to select sample sites, both terrestrial and aquatic, based 
on habitat characteristics. After training, participants designate 
plots — the number is up to each participant — within their chosen 
sites. They visit each plot once, spending approximately 30 minutes 
and collecting data on the distribution of invasive species as well 
as species abundance, percent cover, and mode of reproduction. 

Observations are supported by photos or samples and all data are 
entered online. 

Supplemental training materials are available online, and advanced 
training sessions dealing with specific species also are available. 

Project data are used and interpreted primarily by project scientists 
but are available to the public via the project website. In addition, 
participants are welcome to share their ideas or concerns about 
data collection with the project partners. So far the data have been 
used to inform state legislation on invasive species. For example, 
in Massachusetts, IPANE worked with the Massachusetts Invasive 
Plant Evaluation Project (MIPEG) to develop the Massachusetts 
Prohibited Plant Fist. 

We consider IPANE to be a Collaborative project for three reasons. First, participants select their own 
study sites (although this also occurs in many contributory projects). They are given a set of criteria 
and then use this information to select sites they deem worthy of study. Second, project organizers 
encourage individuals to become involved with curriculum and management projects as evidenced by 
links to those programs at the IPANE web portal (project organizers acknowledge that these linkages 
are still at an early stage in development). Third, and most important, participants are encouraged to 




Participants in the Invasive Plant 
Atlas of New England monitor the 
occurrence of non-native species 
such as this Japanese Barberry. 
Photo courtesy IPANE 
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disseminate the data they colleet, and project organizers work to link interested individuals to partner 
organizations such as the Connecticut Invasive Plants Council, Connecticut Invasive Plant Working 
Group, the Northeastern National Parks Invasive Evaluation Project, MIPEG, and the New England 
Invasive Plant Group (NIPGro). 

Because the original goal of this project was primarily scientific, impact assessment has focused on 
the number of records collected across the region. To date, 11,596 field records have been submitted 
on 5,389 field forms. Clearly the project has developed an early-detection invasive-plant network 
from which participants and scientists can benefit. 

While this project has not been extensively evaluated from an educational perspective, a questionnaire 
focusing on assessing participants’ perceptions of the project has been developed and disseminated 
by project staff. The questions focus on specific aspects of the plants to be studied and the nature of 
the ecological interactions within relevant plant communities. Questions were also asked about the 
helpfulness of different aspects of training sessions; Species identification, methodology and data 
forms, and online data entry. In addition, data and website use were assessed. The questions were 
mostly multiple choice although several yes or no questions were followed with “if not, why” which 
required a write-in answer. 

The results helped project organizers to make decisions about updating the training schedule and 
program. Suggestions were solicited at the end of the questionnaire for additional workshop ideas. 
However, staffing limitations appear to hinder organized assessment of learning, and the survey is not 
administered regularly. In terms of desired learning outcomes, based on the nature of the questions, 
the project seems to focus on increasing awareness, engagement, skills, and attitudes. The training 
seems to focus on helping participants understand the process of how the project data need to be 
collected. As the partnerships develop, project organizers intend to further articulate learning goals 
and outcomes. 

Co-created Projects 

Shermans Creek Conservation Association 

Shermans Creek Conservation Association (SCCA), a group of residents of the Shermans Creek 
Watershed in south-central Pennsylvania, conducts several monitoring projects under the guidance 
and mentorship of ALLARM. SCCA also conducts local educational outreach and collaborates with 
local and state officials to implement sound land use practices (www.shermanscreek.org). Begun in 
1998, SCCA receives a variety of grants from organizations such as the Pennsylvania Department of 
Environmental Protection and the League of Women Voters Water Resources Education Network. 

Shermans Creek Watershed covers approximately 250 square miles of Perry County. The audience 
for SCCA’s activities consists mostly of local residents, who live in small towns and surrounding 
agricultural areas. ALLARM supports SCCA not by setting its goals and activities but by helping 
the group reach the goals that its members have agreed upon. ALLARM provides training, technical 
support, QA/QC, and programmatic and grant-writing assistance. 

The goals of SCCA are several. Scientifically the project aims to monitor the health of Shermans 
Creek Watershed to target critical areas for protection or mitigation and to inform management 
decisions. The project also seeks to help all watershed residents become aware of the watershed’s 
importance to their quality of life and to instill in residents a desire for its protection. In addition, 
SCCA hopes to empower community members to meaningfully participate in making decisions about 
how the watershed is managed. 

At the heart of the project is long-term monitoring. When the project began, its overall objectives 
were to collect at least five years of baseline data on the status of Shermans Creek and its tributaries. 
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to disseminate those data, and to restore and/or 
mitigate areas that were identified as needing 
action. These activities have been realized: About 
five core organizers and 14 additional volunteers 
have collected extensive water quality data from 
stream sites throughout the watershed from 
2001 to the present. In fact, these individuals 
have been engaged in virtually every step of the 
scientific process. They have gathered background 
information and, as part of a six-month study-design 
process, have helped to choose questions for study. 

They have been trained by ALLARM staff to collect 
and analyze samples and then, every month for at 
least one year, each volunteer has used field kits 
to analyze streamwater pH, alkalinity, nitrates, 
dissolved oxygen, and temperature. Volunteers 
also have conducted seasonal visual assessments 
of instream, bank, and riparian zone conditions. 

And at least once per year volunteers have sampled 
macroinvertebrates, which they have identified to 
the level of order. 

After three years of monitoring had occurred throughout the watershed, ALLARM conducted two data 
analysis workshops. Leaders used guided discovery to help participants interpret project data, draw 
conclusions, and identify areas of concern. Through this activity participants learned how scientists 
evaluate evidence and what constitutes sufficient evidence for drawing conclusions. They also learned 
about the challenges involved in understanding seasonal and spatial variability. ALLARM used the 
volunteers’ analyses to compile an extensive report that presented their project findings and made 
recommendations for future action. SCCA disseminated the report to residents and municipal offices 
throughout the watershed (Wilderman 2005). 

Many SCCA volunteers took significant action based on the study results. Actions included visiting 
every municipal office in the watershed to network; developing a watershed conservation plan; helping 
a local borough develop a source water protection plan after a contamination event; and petitioning the 
state for an upgrade of the stream to exceptional value, which would require the state to bestow such 
status on a warm water fishery for the first time in history. (A decision on this request is still pending.) 

In addition to monitoring, SCCA also conducts numerous educational activities including monthly 
meetings, stream cleanups, and “kid days.” These programs involve approximately 100 residents each 
year. SCCA was also involved in a full-year educational project with the local high school in which 
multiple courses used the watershed for the focus of their activities, and students conducted projects 
that they presented at the end of the year at a “Town Meeting.” Finally, the SCCA newsletter goes to 
all watershed households two times per year. Participants contribute significant time to this project. 
About five community leaders each contribute about 50 hours per year in project organization, while 
about 15 other volunteers each contribute about 40 hours per year in monitoring. 

Because participants in SCCA’s monitoring effort have been involved in each step of the scientific 
process, they have developed a remarkable degree of understanding about scientific methodology 
along with excellent data collection skills. By participating in study design workshops, participants 
have been able to see their own questions be translated into research projects, and they have learned 
what kinds of questions can be answered through scientific investigation and what kinds of questions 
are better answered by other means. Study design workshops also have increased participants’ 




Members of the Shermans Creek Conservation 
Association monitor the health of the watershed to 
target critical areas for protection. Photo courtesy 
ALLARM 
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awareness of the need to elearly articulate the intended use of data before data-collection methods 
are chosen. While ALLARM played a major role in the initial study design, SCCA volunteers have 
subsequently redesigned the study using their knowledge of fundamental concepts and techniques, 
indicating significant growth in their understanding of the scientific process (Wilderman 2005). 

SCCA volunteers also have learned to analyze and find stories in their data. This activity, along with 
knowledge of local conditions, has allowed volunteers to play a significant role in shaping project 
reports, targeting critical areas for protection and mitigation, and disseminating this information. 

Of course, not all volunteers have been equally impacted by project participation. ALL ARM staff 
feel that those who have attended the data analysis workshops seem to have learned the most. Many 
of these participants initially felt that the science underpinning the project lay in field sampling and 
data analysis, but have grown to understand that the “real” science lies in study design and the ways 
in which evidence is evaluated to draw conclusions. Participants who have chosen to restrict their 
activities to field sampling and analysis have not learned this valuable lesson. 

The majority of volunteers have fulfilled their commitment of attending one training workshop, 
monitoring one stream each month over one full year, and participating in the QC/QA program. 
Recruitment of volunteers for new activities, such as the rivers conservation plan and the petition 
to upgrade the stream, has been reasonably successful. All volunteers have consistently reported 
enjoying workshops and monitoring activities, and most have made an effort to be involved in new 
workshops as they move through the scientific process of monitoring and data analysis. Most also 
have expressed the sense that project activities are making a useful contribution to knowledge about 
the watershed and have reported an increased sense of community involvement (SCCA president, 
Linda Sieber, personal communication, SCCA semi-annual newsletters). 

Reclam the Bay 

Reclam the Bay (RTB) seeks to engage community volunteers in 
maintaining clean water and restoring shellfish to Barnegat Bay, New 
Jersey (www.reclamthebay.com). The project began in 2005 as an 
educational effort of the New Jersey Ocean County Cooperative Extension 
office. In 2006 participants took ownership of the project, designated 
a 501(c)3, and began raising their own funds. The project involves 
approximately 50 volunteers across a range of ages. To date, these 
individuals have contributed more than 4,700 hours of project time. Some 
individuals participate occasionally while others spend several hours per 
month in project activities. 

Volunteers begin by participating in a 24-hour training course in which 
they learn about the biology, ecology, and rearing of shellfish. After 
completing the course and receiving a certificate, participants find suitable 
rearing locations and build upwellers to rear young shellfish. When 
sufficiently grown, the animals are collected and released into the bay 
where they grow to legal size and can be harvested. Participants collect 
data regarding the rearing system as well as information on water quality 
and biotic data such as growth rate. Scientists use the data for publication, while volunteers use it to 
influence decisions about how the bay is managed. In 2008, 1.4 million clams were planted in two 
estuaries and plans are in place for a third estuary. 

We consider RTB to be co-created because participants are deeply engaged in all aspects of the 
project. For starters, both scientists and participants choose questions for study. In 2009, participants 
will investigate why the growth rates of larval shellfish vary. Participants who wish to gather 
background information are encouraged to do so and share the information with scientists. Both 




Participants in Reclam the 
Bay are helping to restore 
shellfish to Barnegat Bay, 
New Jersey. 

Image courtesy Reclam 
the Bay 
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volunteers and seientists develop hypotheses — while the seientists are seen as experts, both groups 
work together to share ideas. And although data-colleetion methodologies were initially designed by 
projeet seientists, some projeet volunteers have now ereated new systems for rearing young shellfish. 

Project scientists guide the data analysis process, but participants are involved and help to interpret 
data and draw conclusions. Participants and scientists also work together to spread awareness of 
project results and to educate the public about the consequences of shellfish filtering on water quality. 
Participants hold several sessions during the year. Finally, participants and scientists work together to 
develop future questions. 

Assessment of project impact has been limited owing to a paucity of resources. Pre- and post-project 
testing of participants using surveys developed by project staff has shown gains in content knowledge 
(project director, personal communication). In addition, many volunteers are highly engaged, not only 
in rearing shellfish but also in leading educational programming. Project participants have started 
a fundraising foundation that has raised $40,000. They also have developed a popular tourism and 
educational program titled “The Clam Trail” that has raised over $50,000. 

The project also has led to partnerships for disseminating information on bay stewardship. Six national 
and state organizations have been included along with two elementary schools and a high school. 
Volunteers have reached 4,600 individuals through 91 community-based presentations, and they 
provide information to a popular radio personality in an effort to further involve the community. 

Project coordinators feel that measuring impact by assessing community involvement is cost effective, 
and that successful community involvement refiects the successful nature of the project’s educational 
programming. 

The following tables summarize the sample projects, goals, and outcomes. 
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Part 5. Impacts of Public Participation in Scientific Research 

Awareness, knowledge, and/or understanding 

All PPSR projects seem to contribute to awareness, knowledge, and/or understanding of key scientific 
concepts related to the study at hand. This conceptual understanding ranges from purely scientific 
information to environmental issues and regulations. Participants in The Birdhouse Network learned 
facts about bird biology; those in Spotting the Weedy Invasives learned about the ecology of invasive 
plants; participants in the ALLARM acid rain study learned about acid deposition; and those in 
Reclam the Bay learned about the habitat needs of shellfish. Gains in content knowledge are among 
the easiest to measure with project-specific surveys and questionnaires, and are typical for most types 
of informal science education projects and programs. 

But participants in many PPSR projects also gain knowledge of the process of science. Indeed, this is 
one area where PPSR projects have the potential to yield major impacts, particularly Collaborative and 
Co-created projects, which engage participants in project design and data interpretation to a significant 
degree. For example, participants in the salal harvest study. Reclam the Bay, and the Shermans Creek 
Watershed study increased their understanding of the process of science to the point where they 
revised study methods and conducted essentially independent research. And participants in CHEIHO 
learned science processes even though many had never been exposed to science before. 

Even participants in Contributory projects report increased science inquiry skills. One-third of 
participants in Spotting the Weedy Invasives stated that their understanding of science process had 
increased. The Monarch Larvae Monitoring Project newsletter describes volunteers who came up with 
research questions of their own, including “Are some sites better monarch homes?” “Does spotted 
knapweed affect milkweed patch quality?” And “Are deer good for monarchs?” These participants 
would not have asked these questions had they not been part of the project. Some participants in 
The Birdhouse Network used the project database to ask and answer their own bird study questions 
or designed their own projects. And some ALLARM acid rain project participants increased their 
understanding of the concepts of variability and sampling intensity and used this knowledge to 
describe the acid deposition problem through letters to the editor, discussions with government 
representatives, and presentations at community events. 

Participants in most of the Collaborative and Co-created projects, including IPANE, SCCA, the 
salal harvester project, and CHEIHO also seem to have gained knowledge of community structure 
and environmental regulation. Salal harvesters learned that belonging to an organization allowed 
them to communicate more effectively with forest managers than if they worked alone. CHEIHO 
participants learned about the laws that could help them stop more hog farms from being located in 
their communities. Members of the Shermans Creek Conservation Association learned that most land 
use planning decisions in Pennsylvania are made at the local level, so they took a copy of their report 
to every municipality to ensure that their voices would be heard at the table. In addition, when their 
analyses indicated that they were studying a high-quality stream, they used their data to petition the 
state to upgrade the stream’s classification and provide it with better protection. These actions indicate 
that SCCA members gained knowledge of community dynamics and environmental regulation. 

Engagement or interest 

Increasing public engagement in scientific activities is an area at which PPSR projects excel. 
Participants who take part in a project’s full range of activities are deeply engaged in conducting 
science, although they may not always realize this fact. The fact that information collected by citizen 
scientists have yielded some of the largest databases available about the distributions and abundances 
of plants and animals on our planet showcases the level of engagement of volunteers in gathering 
information to further the cause of science and environmental management. Indeed, enlisting people 
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into PPSR projects is probably one of the most expedient methods for informal science educators to 
engage people in science in a fun and meaningful way. 

PPSR projects provide opportunities for people to develop interest and engagement by either trying 
something new or by expanding previously existing interests. For example, when a bird watcher 
begins to report her bird sightings or an angler helps to monitor the quality of a favorite fishing 
stream, PPSR helps individuals see that their interests can connect to science, and that learning 
science can enhance their enjoyment of their interests. The different PPSR models allow varied levels 
and types of engagement so that individuals with varying levels of expertise can either develop a new 
interest or take deeper steps in their practice or engagement. The fact that PPSR projects are often 
long-lived and frequently cross large geographic boundaries provides opportunities for continuity over 
time and for people to participate, take breaks, and then return. 

Indeed, many participants in PPSR projects return year after year. Individual participants in The 
Birdhouse Network, ALLARM’s acid rain and watershed projects, IPANE, and Reclam the Bay 
have gathered data for many years. Many project directors report a small cadre of participants who 
engage at particularly intense levels and, in certain cases, take a leadership role in the project. While 
outcomes of this intense level of participation vary, a notable example is that of participants in Reclam 
the Bay, who raised nearly $100,000 and took project information to thousands of individuals in their 
community. Some CHEIHO participants took their data to state regulatory agencies to revise land 
use laws around hog farm development, and participants in IPANE took their data to members of 
management agencies, who used the information while creating lists of threatened and endangered 
species. 

Some PPSR projects also help to build community. Large-scale Contributory projects such as The 
Birdhouse Network tend to position themselves around virtual communities, while Collaborative or 
Co-created projects with face-to-face interactions are effective at building “real-world” communities. 

Skills 

PPSR projects are excellent for developing science-related skills. Participants in most projects 
increase their ability to identify organisms, to use measurement instruments, to collect field data 
following specific protocols, and to sample consistently over time. Eor example, 95% of individuals 
who started the ALLARM Acid rain project passed the quality assurance test, and in Spotting the 
Weedy Invasives, volunteers were able to correctly indentify 90% of common invasives, with 
variance in participant accuracy measured as similar to that of professionals (project director, personal 
communication). Thousands of individuals have submitted data on nesting birds, the distribution and 
abundance of monarch butterflies, patterns of water quality, and the distribution and abundance of 
invasive plants. Eurthermore, these data have been of sufficient quality to allow scientific analyses and 
to be published in peer-reviewed scientific publications (see www.citizenscience.org for a growing 
list). 

Participants in data analysis workshops gain skills in reading and interpreting graphs, drawing 
conclusions from evidence, and raising additional questions as a basis for new study designs. In 
the ALLARM data interpretation workshops, volunteers learned basic concepts of causes, effects, 
and interactions of water pollutants through a self-discovery exercise with a virtual watershed in 
preparation for informed interpretation of their own data. Participants in Collaborative and Co-created 
projects also increased their ability to identify and distinguish habitats, to identify and select study 
sites, to weigh the pros and cons of various research design and data collection methods, and to 
communicate their results to the public. 

However, helping participants learn the skills that take them beyond data collection and into study 
design, data analysis, and data interpretation is not an easy task. Typically it requires a significant 
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degree of individualized attention. For example, volunteers do not always agree about the most 
important concerns to be addressed, so the study design process often becomes focused on developing 
consensus on priorities. Also, volunteers usually need help in focusing their concerns on questions 
that can be answered within an appropriate time frame and with available resources. Once established, 
however, questions that volunteers have raised themselves provide a high level of motivation for 
project participation. 

Helping volunteers learn to analyze and interpret data is also time consuming, and “short cuts” do not 
generally work. For example, ALLARM has used a training model where scientists present completed 
project reports to participants and explain how their data were analyzed and the significance of their 
findings. In these cases, the volunteers invariably claim that they do not understand the patterns well 
enough to disseminate them or to discuss the results with others. On the other hand, when ALLARM 
staff train volunteers to find stories in the data themselves, the volunteers are eager and able to 
disseminate results. 

Attitudes 

Few instances where PPSR project participation has affected attitudes toward science have been 
documented. Only one of the projects that we investigated. The Birdhouse Network, formally 
attempted to measure attitude change, with no change demonstrated. This may not be surprising, as 
we kn ow that participants in many PPSR projects are aheady interested in science. Nevertheless, 
individuals who learn to function as scientists, or at least to understand how scientists work, could be 
expected to increase their already positive attitudes toward science. For example, parents stated that 
their children felt like scientists after participating in the MLMP 

We do see evidence that Co-created projects that are initiated to meet specific community needs 
can draw concerned citizens into the scientific process who might not otherwise be involved in 
science-related activities. This was particularly true for the salal harvester project and CHEIHO. Such 
participants are likely to experience significant changes in their attitudes toward science, although 
whether such change will take place, and whether change will be positive or negative, remains to be 
studied. 

Indeed, many participants probably learn that scientific investigation can be tedious. But even 
individuals who decide that they do not want to become scientists may become better consumers of 
science as both their understanding and attitudes change. 

Behaviors 

Although behavior change is often considered to be the “Holy Grail” among project impacts, it is the 
least studied and understood owing to the challenges of determining causal relationships in varied 
learning environments. However, anecdotal evidence does suggest that participation in PPSR projects 
can change behaviors. First of all, the projects cause people to take part in scientific research, which 
is a huge behavior change for many! Participation also can lead to other types of behavior change. 

For example, participants in TBN showed increased “pro-bird” behaviors such as improving their 
habitat for wildlife. Some participants in Spotting the Weedy Invasives said that they had begun to 
take environmental concerns into account when purchasing plants, and 40% said that they had begun 
to notice invasives in the environment. Members of the ALL ARM acid rain project were so desirous 
of increasing their water quality monitoring activities that the organization broadened its focus and 
changed its name. And harvesters who participated in the salal research project demonstrated an 
increase in involvement in the NRHA and in the harvester community as a whole. In the SCCA 
project, volunteers became more engaged in community politics, and more confident about asking for 
a place at the table in decision making about future land use for the watershed. 
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Part 6. Opportunities for the ISE Field 

Many types of informal science education programs and projects are valuable for increasing public 
understanding of science content and concepts. Many types of projects also can help to change 
attitudes and behaviors related to science and the environment (National Research Council 2009). 
Projects that directly involve members of the public in scientific research seem particularly suitable 
for increasing participants’ understanding of science process, creating deep engagement in science, 
and developing multiple scientific skills including observation, study design, sampling, data 
collection, data analysis, and drawing conclusions from evidence. PPSR projects may also be valuable 
in changing behaviors and attitudes toward science, but indicators that can measure such impact will 
need to be developed to draw decisive conclusions with respect to these impacts. 

We suggest that projects focusing on Public Participation in Scientific Research represent an emerging 
field of informal science education that is ripe for further development. We urge developers of ISE 
programs to consider PPSR approaches in their work. This could involve 

1) creating new PPSR projects 

2) enhancing PPSR projects already underway 

3) adding PPSR elements to other types of ISE programs, such as exhibitions 

4) enhancing research and evaluation of PPSR including best practices for developing different 
models and understanding their impacts on participants, researchers, and communities. 

Creating new PPSR projects 

Although hundreds of PPSR projects are now under way, both large and small, room for new projects 
exists in three areas; 

1) Projects designed to study new scientific questions 

2) Projects designed to engage new audiences 

3) Projects designed to test new or enhanced PPSR models. 

The natural world is full of questions whose answers truly require a PPSR approach. The number 
of published scientific papers based on citizen-collected data is increasing each year (see www. 
citizenscience.org). Many more projects could be created that will appeal to the increasing numbers 
of amateur naturalists and stargazers who are interested in lending their brains to science. Developers 
should be aware, though, that coming up with questions that are appropriate for PPSR is not a simple 
task, especially if the project desires to reach a wide audience. Projects for which data collection 
relies on basic skills, such as counting a few species of birds at feeders or determining the pH of a 
water sample, are generally more appropriate than projects that require high levels of skill, at least 
for projects that follow the Contributory model. Also, projects that are based on popular hobbies like 
watching birds generally draw more participants than projects that are not (Bonney et al. in press). 

Prom an educational perspective, however, developing projects to answer new scientific questions 
may not be the most strategic approach for the ISE field unless new projects strive to enhance existing 
PPSR strategies or reach truly new audiences. Most of the projects studied for this report, especially 
those labeled Contributory, seem to reach audiences already knowledgeable about science and the 
environment. While such audiences can always learn more — especially advanced science skills — the 
case studies that focus on Collaborative and Co-created models have shown that non-typical audiences 
can be reached with topics and approaches that speak to specific personal and/or community needs 
and interests. 
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To reach new audiences, however, significant research into motivations for members of the public to 
understand and participate in research will be needed. Also required is research into the ways in which 
individuals perceive themselves or can develop identities as scientists, potential scientists, and critical 
thinkers. To date, very little front-end evaluation has been carried out to assess community desire, 
interest, or perceived relevance of PPSR, especially for contributory projects. In fact, we are aware of 
only two studies, both NSF-funded planning grants, that have investigated perceptions of science and 
PPSR possibilities for audiences who are not traditionally involved in ISE programming: (Garibay, 
2004; Garibay, 2009). Both of these studies have revealed that new audiences (in these cases. 

Latinos) will be most interested in projects that are highly relevant to pressing needs and issues for a 
community. Further, the number of individuals who will be involved in such projects may not be high, 
at least not at first. This fact is evident in examining Co-created projects, which in most cases do result 
from community-identified needs, and which can reach diverse audiences at least in small numbers. 

We suggest that new projects test Collaborative and Co-created approaches to PPSR. One idea that 
seems worthy of further examination is the approach taken by Reclam the Bay, the Salal Harvest 
Sustainability project, and CHEIHO of working with a “core” of community members who collaborate 
closely with scientists as part of a research team and participate to a high degree, and a secondary level 
of participation by a larger number of community members who primarily contribute to data collection 
and possibly study design. This model of “peripheral participation,” developed by Lave and Wenger 
(1991), has been widely used to move people from peripheral to core participation in a variety of 
efforts. For projects that wish to use a Co-created approach, key factors are building trust and mutual 
respect between scientists and participants, having an ethical commitment to protect participants from 
harmful consequences of participating in research, and creating transparency in the entire decision- 
making process at every step of the research project. 

Enhancing PPSR projects already underway 

Our case studies suggest that projects which involve members of the public in the largest number 
of steps or categories of research, those that we have categorized as Collaborative or Co-created, 
yield the greatest impacts in terms of understanding science process, developing skills of scientific 
investigation, and changing participants’ behaviors toward science and/or the environment. Indeed, it 
stands to reason that people who have participated intensively in a research or monitoring project will 
be that much better consumers of science, will be more active in decision making around science and 
the environment in their communities, and will change their behavior toward the scientific problem 
being investigated more substantially, than individuals who are involved peripherally in a project. 
Therefore, we believe there is tremendous potential to enhance projects that are already under way 
on several fronts. In particular, we suggest that many projects could be expanded to involve more 
individuals in more aspects of science inquiry. 

For instance. Contributory projects can be expanded into Collaborative models, at least for some 
project participants. One idea is to involve participants in the study design process. In the case of 
SCCA, participants dealt with the study design process more easily after the initial study design was 
completed. They increased their confidence and direction, and were able to mark their progress in a 
systematic manner. They also were encouraged to understand the iterative nature of designing a study, 
informed by feedback from monitoring activities in the field. However, helping individuals understand 
how research and scientific data are used and applied to policy and management must be deliberative 
and facilitated. Learners must not only be exposed to ideas, they must understand how experts think 
about those ideas. 

Indeed, informal science educators are recommending that the science behind ISE projects be both 
explicit and refiective (Akindehin 1988; Crawford et al. 2005), allowing learners the opportunity to 
refiect upon and refine ideas, and to practice and make mistakes. Further, given that experts have a 
hierarchically organized scheme by which they reason about scientific problems (Bransford et al. 
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1999), participants need to be exposed to educational experiences that not only teach about particular 
concepts but also about ways to think about and practice these concepts. More research and testing 
will be required to learn how to provide such opportunities for large audiences, that is, to bring data 
analysis and interpretation, hypothesis testing, and discussion of results to projects that have started 
with a contributory design. As MLMP already has started to do, participants could see videos or 
online tutorials that help them develop questions and projects of their own that would take their data 
collection activities further and deeper. Other ideas include holding regional training workshops 
where participants can learn skills of data interpretation and analysis, making data more available and 
providing tools for simple visualization and analysis, or providing the opportunity for participants to 
develop their own questions and contribute their findings to participant blogs or online newsletters as 
occurred in TBN. 

At the same time. Collaborative and Co-created projects — in many cases, projects that have been 
started more to meet community needs than for educational reasons — can learn many lessons from 
successful Contributory projects. For example, water quality monitoring projects could enhance 
educational support materials to help participants learn more about aquatic organisms, such as online 
videos or identification keys. Such projects also could work together to design and develop regional 
and national databases that would allow participants to compare data and findings across larger 
regions or over long spans of time. Such studies could prove particularly valuable for studying large- 
scale issues such as the effects of global warming on water quality. They also would help participants 
place their research in a global context. 

Our suggestion that the ISE field take a hard look at enhancing projects that are already underway 
does not preclude an important role for organizations or institutions that have not previously been 
involved in PPSR efforts. On the contrary, we believe that it increases such opportunities and 
encourages the development of new partnerships. Almost any institution could find an existing 
project, such as NestWatch, MLMP, a water quality monitoring project, or any of the other dozens of 
projects listed at the Citizen Science Central website (www.citizenscience.org) and adapt or modify it 
for local or regional use, for example, by adding workshops to take participants as deeply as possible 
into all science process activities surrounding that project. This approach would be one of the easiest 
and most effective methods of adding Collaborative or Co-created approaches to projects that already 
provide significant infrastructure for PPSR. In other words, it would leverage previous work and allow 
the testing of new approaches to expose existing participants to more advanced scientific concepts and 
skills and tailor existing projects to new audiences. 

Another idea for engaging new audiences is to employ the newest in Web 2.0 technologies to existing 
projects. If a major goal of ISE is to engage younger audiences, then projects must be relevant and 
current for such audiences. As an initial first step, projects should provide alternate avenues to reach 
new audiences by building online social presence through FaceBook, Twitter, My Space, and a slew 
of other social networking sites. Creating an online presence, however, does not necessarily equate 
to creating an online community. More research on best practices for online learning and community 
building should be pursued and broadly disseminated. 

Adding PPSR to other types of ISE projects 

Examination of the NSE portfolio shows that relatively few full-fiedged PPSR projects have been 
funded to date, although several have started in the last few years. More common in the portfolio are 
PPSR “add-ons,” that is, citizen science projects that are associated with other projects such as films 
or exhibits. We suggest that more closely integrating PPSR efforts into other types of ISE projects 
would represent strategic investment by the field. However, we suspect that many project developers 
will underestimate the amount of funding and effort that is required to develop a PPSR project, and 
we predict that many “side projects” will not yield demonstrable impacts unless they build on PPSR 
projects and networks that already exist. 
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Again, we suggest that ISE projects seeking to deepen their impact through inclusion of PPSR 
techniques consider partnering with pre-existing PPSR projects. Indeed, such partnerships would 
benefit both programs. For example, a traveling exhibition about monarch butterflies could be used to 
increase interest and participation in the MLMP, while the PPSR project would provide a meaningful, 
long-term, immersion experience for visitors who become interested in monarchs and want to take 
their interest further. 

Enhancing research and evaluation surrounding models of PPSR and their impacts 

Many questions remain to be answered about the potential for PPSR projects to contribute 
significantly to the ISE field. An important question is what motivates participants to participate in 
PPSR projects, which raises a number of additional questions about personal and extrinsic motivators, 
altruistic intentions, and the notion that projects may be tiered or serve as gateways for other projects 
to embrace participants who join projects for different reasons. 

Many questions also remain about the learning implications of various types of PPSR projects. What 
is the nature of questions that enable participants to not only collect data but also to refiect on their 
experiences? Do participants, or potential participants, prefer to become involved with questions that 
deal with basic science, that resolve issues, that are simply fun, or all of the above? To what extent do 
participants gain from projects because they help to shape them? 

Other questions relate to the overall impacts of PPSR participation, including participation in areas of 
inquiry that have not been well studied such as modeling, data visualization, and data dissemination. 
Also important to understand are the behavioral impacts of PPSR. Does participating in a national bird 
count cause consumers to purchase shade grown coffee? Does spending time collecting water samples 
lead to reduced water consumption activities? Indeed, understanding the impacts of PPSR could push 
the boundaries of what we currently define as learning in the realm of science, including learning that 
affects participants’ lives in a very broad sense. 

Equally important is examining the learning impacts for scientists involved in PPSR efforts. What are 
they learning in terms of science knowledge, science process, and attitudes about science? This type of 
research has not been done at all as far as we can determine. 
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Part 7. Conclusions 



Public Participation in Scientific Research projects, especially those labeled as citizen science, are 
growing in numbers every day. A quiek Google seareh will yield hundreds of projects under way 
across North America and beyond, each one hungry for volunteers. Considering the potential for such 
projects to contribute high-quality data for seientific monitoring and research, and to help participants 
develop a greater understanding of science — both content and process — the PPSR “movement” seems 
well justified. 

At the same time, the proeess of studying and understanding the best ways to develop, implement, 
and evaluate PPSR projects is just beginning. Based on a review of a handful of projects representing 
a range of PPSR models, we have suggested several ideas for PPSR project development and many 
areas in which we believe that the field can grow as a discipline. We believe that, if project developers 
try some of the ideas suggested here, the field will expand in new and compelling directions. Many 
opportunities exist for continued study of the PPSR eoncept, including additional project reviews that 
might uncover more robust conclusions about project impacts, and which could suggest additional 
avenues for project development. In particular, we urge continued research into project designs that 
will harness the strengths of each PPSR model, merging model elements to yield the greatest learning 
impacts. 

We also believe that the field must do a far better job of articulating learning goals and desired 
outcomes than most PPSR projects have done so far. In addition, the field must develop more 
effective and comprehensive methods of impaet evaluation. At a June 2007 workshop, “Developing 
a Citizen Science Toolkit,” funded by the National Science Foundation and held at the Cornell Lab 
of Ornithology, more than 50 individuals gathered to compare notes on developing “best practiees” 
for citizen science project development. Tools and tips were offered for essentially every area 
except project evaluation. Many participants noted that evaluation is the most challenging and least 
understood step in the process of project development, which is one reason why the meta-analysis that 
we conducted for this report yielded so few robust evaluation reports. 

We conclude that developing, validating, and disseminating evaluation strategies would significantly 
enhance the impact of PPSR projects and the ISE field as a whole. Such strategies and tools are 
needed to guide project development. They also would provide ISE professionals with methods for 
measuring baseline scientific and environmental literacy and for documenting ehanges in literacy that 
could be attributable to PPSR project participation. We urge the field to begin developing, testing, and 
employing more robust evaluation methods and to make a major effort to share project results, both 
positive and otherwise, to help the PPSR field eontinue to grow and mature. We hope that the rubric 
provided in this report will help to move this effort forward. 
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Appendix 

Public Participation in Research Projects funded by the National 
Science Foundation since 2000 

Informal Science Education Program 

Although hundreds if not thousands of citizen science and related PPSR projects are under way today, 
relatively few have been funded by the Informal Science Education (ISE) program at the National 
Science Eoundation (NSE). Most of the projects that have been funded by the ISE program and which 
are fully operational have been developed by the Cornell Eab of Ornithology (CEO). We selected one 
NSE-funded CEO project, The Birdhouse Network, for review in this report. The only non-CEO PPSR 
projects that have been funded by ISE and which are “completed” — as far as we can determine — are 
the Monarch Larva Monitoring Project developed by the University of Minnesota and CoCoRaHS, 

The Community Collaborative Rain, Hail, and Snow Network developed at Colorado State University. 
Both of these projects also are reviewed in this report. 

Three additional projects have been funded recently by the ISE program and their development 
is now underway. These are Citizen Science Laboratory at Hacienda la Esperanza, developed by 
the Conservation Trust of Puerto Rico; Have You Spotted Me? Learning Lessons by Looking for 
Ladybugs, developed by the Department of Entomology at Cornell University; and Communicating 
Climate Change, a network of citizen science projects being organized through the Association of 
Science-Technology Centers. 

In addition, one recently funded exhibition project includes a citizen science component: Seasons of 
Change: Signs of Climate Change in New England and North Carolina, which is under development 
by Brown University. 

We provide here a brief overview of ISE-funded PPSR projects. 

Public Participation in Ornithology: An Introduction to Environmental Research (DRL- 
9155700). Cornell Lab of Ornithology: 1992-1996. 

This project launched the Citizen Science Program at the Cornell Lab of Ornithology. It funded the 
development of Project PigeonWatch (ongoing; www.birds.cornell.edu/pigeonwatch) and Project 
Tanager (now Birds in Eorested Landscapes; www.birds.cornell.edu/bfl); major enhancements to 
Project FeederWatch (ongoing; www.birds.cornell.edu/pfw); and a national Seed Preference Test. 
PigeonWatch involves city dwellers in counting the kinds and numbers of different color types of 
pigeons throughout the United States; Birds in Forested Landscapes allows birders to help study 
the habitat requirements of many species of North American birds; and Project FeederWatch is a 
fall/winter count of the kinds and numbers of birds that visit backyard feeders in the United States and 
Canada. 

The Cornell Nest Box Network (DRL-9627280). Cornell Lab of Ornithology: 1996-2001. 

The Cornell Nest Box Network was renamed The Birdhouse Network (TBN) and is fully described in 
the sample projects section of this report. TBN was absorbed into NestWatch in 2007 (see below). 

Parents Involved/Pigeons Everywhere (DRL-9802248/0926149). Cornell Lab of Ornithology: 
1998-2004. 

This Parental Involvement grant developed major enhancements to Project PigeonWatch (www.birds. 
cornell.edu/pigeonwatch) though collaborations with a network of science museums, science centers, 
and nature centers. 
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Citizen Science Online (DRL-0087760). Cornell Lab of Ornithology: 2001-2006. 



This project funded the development of two ongoing projects: the Great Backyard Bird Count (www. 
birdsource.org/gbbc) and eBird (www.eBird.org). The Great Backyard Bird Count is a one-weekend 
count of backyard birds that takes place throughout North American each February, and eBird allows 
birders to report any bird they see, any time, any place throughout most of the western hemisphere. 
Both projects are cosponsored by the National Audubon Society. 

Monarch Butterfly Larval Monitoring: A Nationwide Citizen Science Initiative (DRL-0 104600). 
University of Minnesota: 2001-2006. 

This project, now called the Monarch Larva Monitoring Project (www.mlmp.org), is growing every 
year and is fully described in the sample project section of this report. 

Birds in the Hood (DRL-0125633). Cornell Lab of Ornithology: 2002-2007. 

This award funded development of Urban Bird Studies, essentially a simplified version of eBird 
appropriate for urban audiences. The project evolved into the current Celebrate Urban Birds! (www. 
birds . c omell . edu/celebration) . 

Bringing CoCoRaHS to the Central Great Plains: an Informal Science Education Project for 
Rural Communities (DRL-0229723). Colorado State University: 2003-2006. 

The Community Collaborative Rain, Hail, and Snow Study involves participants in collecting 
precipitation data across the country and is fully described in the sample projects section of this 
report. 

Citizen Science Laboratory at Hacienda La Esperanza Reserve (DRL-0327374). Conservation 
Trust of Puerto Rico: 2007-current. 

The Conservation Trust of Puerto Rico, in collaboration with the Interdisciplinary Center for Coastal 
Studies, New York Botanical Garden, Puerto Rico Youth at Risk, Boy Scouts, and the Cornell Lab 
of Ornithology is developing a comprehensive citizen science program for involving the public in 
researching archeology and human impacts on local ecosystems; conservation and restoration of 
wetlands; seed dispersal by bats and birds; ecology of coastal land crabs; fiowering and fruiting 
patterns of local fiora; and shoreline and costal processes at one of the Trust’s premier reserves. 

Project NestWatch (DRL-0540185). Cornell Lab of Omithology/Smithsonian Migratory Bird 
Center: 2006-current. 

NestWatch involves participants in collecting data about bird breeding biology across North America. 
It also involves a virtual citizen science project, called Camclickr, for which participants gather 
data from online images provided by a series of NestCams located in several different states (www. 
nestwatch.org). The former Birdhouse Network, described in the sample project section of this report, 
was absorbed into NestWatch in 2007. 

Seasons of Change: Signs of Climate Change in New England and North Carolina (DRL- 
0714744). Brown University: 2007-current. 

Seasons of Change will be a traveling exhibition highlighting how regional iconic “harbingers” 
are related to climate change — for example, the impacts of a changing climate on the maple syrup 
industry in New England and shifts in bird migration patterns in North Carolina. The project will 
include a citizen science program for installations that can provide outdoor venues. 

Have You Spotted Me? Learning Lessons by Looking for Ladybugs (DRL-0741738). Cornell 
University Department of Entomology: 2008-current. 
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Now called the Lost Ladybug Project (www.ladybug.org), this project involves rural ehildren across 
the continent in searehing for endangered and threatened species of ladybugs. Participants submit data 
in the form of digital photos. 

Communicating Climate Change (DRL-0813135). Assoeiation of Seience-Teehnology Centers: 
2008-eurrent. 

The Association of Science-Technology Centers (ASTC), in eollaboration with the Yale Project on 
Climate Change and the Cornell Lab of Ornithology, is developing a network of science eenters that 
are using citizen scienee as a means of helping visitors understand local impacts of climate ehange 
(http://astc.org/iglo/c3). The projeet is beginning with 12 seience eenters spanning the United States 
from the east coast to Hawaii. Additional partners include the University Corporation for Atmospheric 
Research, the American Geophysical Union, and NOAA. 

Other NSF programs 

A small amount of work in PPSR has been eonducted with funding provided by NSF programs outside 
of ISE. Providing a complete list is beyond the seope of this report. Three examples, however, inelude: 

Community Dynamics of an Emergent Pathogen: Intrinsic vs Extrinsic Mechanisms (Emerging 
Erontiers 0622705). Cornell University: 2006-eurrent. 

This project explores the effeets on disease dynamics of multiple hosts that differ in eompetenee and 
transmission routes. Some of the data for this researeh eome from a citizen scienee projeet run by the 
Cornell Lab of Ornithology called the House Finch Disease Survey (www.birds.eornell.edu/hofi). 

CI-Team Implementation Project: Using the GODM Cyherinfrastructure to Involve Citizen 
Scientists in Moving from Data Isolation to Data Integration (Office of Cyherinfrastructure 
0632613). Colorado State University: 2006-present. 

The aim of this project is to empower citizen scientists to use an existing cyherinfrastructure to 
digitally collect, input, integrate, and analyze data on the distribution of non-native plants and animals. 

Citizen Science and Sea Turtle Conservation: Critical Perspectives from Social Studies of 
Science (Science, Technology, and Society 0724608). Duke University: 2007-current. 

This research, a collaboration between Duke University, Nicholas School of Environment and 
Earth Sciences, and the North Carolina Wildlife Resources Commission (NCWRC), examines the 
relationship between volunteers collecting data for the North Carolina Sea Turtle Project and the 
scientific establishment to which they contribute their information. 
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